A  COMPARATIVE  ANALYSIS  OF  THE  COST  ESTIMATING  ERROR  RISK 
ASSOCIATED  WITH  FLYAWAY  COSTS  VERSUS  INDIVIDUAL 
COMPONENTS  OF  AIRCRAFT 

THESIS 


Jonathan  D.  Ritschel,  Captain,  USAF 

AFIT/GCA/ENV/03-08 

DEPARTMENT  OF  THE  AIR  FORCE 
AIR  UNIVERSITY 

AIR  FORCE  INSTITUTE  OF  TECHNOLOGY 


Wright-Patterson  Air  Force  Base,  Ohio 


APPROVED  FOR  PUBFIC  RELEASE;  DISTRIBUTION  UNLIMITED. 


The  views  expressed  in  this  thesis  are  those  of  the  author  and  do  not  reflect  the  official 
policy  or  position  of  the  United  States  Air  Force,  Department  of  Defense,  or  the  U.  S. 
Government. 


AFIT/GCA/ENV/03-08 


A  COMPARATIVE  ANALYSIS  OF  THE  COST  ESTIMATING  ERROR  RISK 
ASSOCIATED  WITH  FLYAWAY  COSTS  VERSUS  INDIVIDUAL  COMPONENTS 

OF  AIRCRAFT 


THESIS 


Presented  to  the  Faculty 

Department  of  Systems  and  Engineering  Management 
Graduate  School  of  Engineering  and  Management 
Air  Force  Institute  of  Technology 
Air  University 

Air  Education  and  Training  Command 
In  Partial  Fulfillment  of  the  Requirements  for  the 
Degree  of  Master  of  Science  in  Acquisition  Management 


Jonathan  D.  Ritschel,  BBA 
Captain,  USAF 

March  2003 


APPROVED  FOR  PUBLIC  RELEASE;  DISTRIBUTION  UNLIMITED. 


AFIT/GCA/ENV/03-08 


A  COMPARATIVE  ANALYSIS  OF  THE  COST  ESTIMATING  ERROR  RISK 
ASSOCIATED  WITH  FLYAWAY  COSTS  VERSUS  INDIVIDUAL  COMPONENTS 

OF  AIRCRAFT 


Jonathan  D.  Ritschel,  BBA 
Captain,  USAF 


Approved: 


_ //signed// _ 

Michael  A.  Greiner,  Maj,  USAF  (Chainnan) 


7  Feb  2003 
date 


_ //signed// _ 

Michael  J.  Seibel  (Member) 


7  Feb  2003 
date 


7  Feb  2003 


_ //signed// _ 

Daniel  E.  Reynolds  (Member) 


date 


Acknowledgements 


I  would  like  to  express  my  sincere  appreciation  to  my  faculty  advisor.  Major  Mike 
Greiner  for  his  outstanding  guidance  and  support  throughout  this  thesis  effort.  His 
direction  and  patience  have  been  critical  to  the  success  of  this  thesis.  I  would  also  like  to 
thank  my  other  committee  members,  Mr.  Mike  Seibel  and  Mr.  Dan  Reynolds.  Both  gave 
very  generously  of  their  time.  Mr.  Seibel’s  insight  as  an  aircraft  cost  analyst  expert  was 
critical  to  ensuring  this  thesis  effort  addressed  the  problem  in  an  acceptable  context  for 
the  aircraft  acquisition  environment.  Additionally,  Mr.  Reynolds  analytic  skills  as  a 
statistician  were  critical  to  ensuring  my  statistical  analysis  met  the  rigors  required. 

Finally,  I  would  like  to  thank  my  wife  for  her  understanding,  patience,  and  support  during 
this  time  consuming  process. 


Jonathan  D.  Ritschel 


Table  of  Contents 


Page 

Acknowledgements . iv 

List  of  Figures . vii 

List  of  Tables . x 

Abstract . xii 

I.  Introduction . 1 

Background . 1 

Problem  Statement . 4 

Research  Objectives . 5 

Methodology . 6 

Scope  and  Limitations . 7 

Thesis  Overview . 8 

II.  Literature  Review . 9 

Chapter  Overview . 9 

Work  Breakdown  Structure . 9 

WBS  Terminology  Clarification . 13 

Cost  Estimating  Relationships . 13 

Aircraft  Estimation  Techniques . 18 

Parametric  Estimation . 19 

Grass  Roots  Estimation . 22 

Previous  Research . 24 

Large,  Campbell,  and  Cates:  1976 . 24 

Hess  and  Romanoff:  1987 . 26 

Rester,  Rogers,  and  Hess:  1991 . 28 

Younossi,  Kennedy,  and  Graser:  2001 . 29 

Summary . 30 

III.  Methodology . 31 

Chapter  Overview . 3 1 

Methodology  Overview . 31 

Data . 3 1 

Data  Limitations . 33 

Variables . 33 

Regression . 34 


v 


Page 


Monte  Carlo  Simulation . 37 

Bootstrap . 38 

Application  of  The  Bootstrap,  Monte  Carlo,  and  Crystal  Ball . 39 

Drawing  Conclusions . 40 

Summary . 40 

IV.  Findings  and  Discussion . 42 

Chapter  Overview . 42 

Multiple  Regression  Models  for  the  Fighter  Category . 42 

Data . 42 

Preliminary  Modeling  Problem . 43 

Basic  Airframe  Model . 45 

Other  Air  Vehicle  Model . 54 

Flyaway  Cost  Model . 62 

Bootstrap  Comparison  for  Flyaway  Versus  Basic  Airframe . 68 

Bootstrap  Comparison  for  Flyaway  Versus  Other  Air  Vehicle . 70 

Multiple  Regression  Models  for  the  Inter-theatre  Airlift  Category . 72 

Data . 72 

Basic  Airframe  Model . 72 

Other  Air  Vehicle  Model . 80 

Flyaway  Cost  Model . 86 

Bootstrap  Comparison  for  Flyaway  Versus  Basic  Airframe . 91 

Bootstrap  Comparison  for  Flyaway  Versus  Other  Air  Vehicle . 94 

Summary . 95 

V.  Conclusions . 97 

Importance  of  Findings . 97 

Limitations . 97 

Discussion  of  Results . 98 

Practical  versus  Statistical  Significance . 100 

Future  Research . 100 

Appendix  A.  Basic  Airframe  Regression  Model  at  Quantity  100 . 102 

Appendix  B:  Bootstrap  Data  for  Fighter  Category . 105 

Appendix  C:  Bootstrap  Data  for  Inter- theatre  Airlift  Category . 110 

Bibliography . 115 

Vita . 119 


vi 


List  of  Figures 


Figure  Page 

Figure  1.  AFCAA  Personnel  Changes  (Deputy,  1999:53) . 2 

Figure  2.  CER  Development  Process  (Department  of  Defense  (c),  1995:111-10) . 17 

Figure  3.  Composite  Material  Usage  (Resetar,  Rogers,  and  Hess,  1991:3) . 29 

Figure  4.  Monte  Carlo  Simulation  of  a  Physical  System  (Nelson,  1997) . 37 

Figure  5.  Constructing  Bootstrap  Samples  (Stine,  2002:Chapter  1:10) . 39 

Figure  6.  Continuous  Data  Histogram  for  Basic  Airframe . 45 

Figure  7.  Assumption  Violation  Plots  for  Basic  Airframe . 46 

Figure  8.  Cook's  D  Influence  for  Basic  Airframe . 48 

Figure  9.  Normality  Histogram  for  Basic  Airframe . 50 

Figure  10.  Autocorrelation  Function  for  Basic  Airframe . 51 

Figure  1 1 .  Residual  by  Predicted  Plot  for  Basic  Airframe . 52 

Figure  12.  Plot  of  Residuals  for  Basic  Airframe . 54 

Figure  13.  Continuous  Data  Histogram  for  Other  Air  Vehicle . 54 

Figure  14.  Cook's  D  Influence  for  Other  Air  Vehicle . 56 

Figure  15.  Revised  Cook's  D  Influence  for  Other  Air  Vehicle . 58 

Figure  16.  Normality  Histogram  for  Other  Air  Vehicle . 58 

Figure  17.  Autocorrelation  Function  for  Other  Air  Vehicle . 60 

Figure  18.  Residual  by  Predicted  Plot  for  Other  Air  Vehicle . 61 

Figure  19.  Plot  of  Residuals  for  Other  Air  Vehicle . 62 

vii 


Page 


Figure  20.  Continuous  Data  Histogram  for  Flyaway  Cost . 62 

Figure  2 1 .  Cook's  D  Influence  for  Flyaway  Cost . 63 

Figure  22.  Normality  Histogram  for  Flyaway  Cost . 65 

Figure  23.  Autocorrelation  Function  for  Flyaway  Cost . 66 

Figure  24.  Residual  by  Predicted  Plot  for  Flyaway  Cost . 67 

Figure  25.  Plot  of  Residuals  for  Flyaway  Cost . 68 

Figure  26.  Frequency  Chart  of  Flyaway  Bootstrap  Data . 69 

Figure  27.  Frequency  Chart  of  Basic  Airframe  Bootstrap  Data . 69 

Figure  28.  Differenced  Flyaway  vs.  Basic  Airframe  Data . 70 

Figure  29.  Frequency  Chart  of  Other  Air  Vehicle  Bootstrap  Data . 7 1 

Figure  30.  Differenced  Flyaway  vs.  Other  Air  Vehicle . 71 

Figure  3 1 .  Continuous  Data  Histogram  for  Basic  Airframe . 73 

Figure  32.  Cook's  D  Influence  for  Basic  Airframe . 74 

Figure  33.  Revised  Cook's  D  Influence  for  Basic  Airframe . 75 

Figure  34.  Normality  Histogram  for  Basic  Airframe . 77 

Figure  35.  Autocorrelation  Function  for  Basic  Airframe . 78 

Figure  36.  Residual  by  Predicted  for  Basic  Airframe . 79 

Figure  37.  Plot  of  Residuals  for  Basic  Airframe . 80 

Figure  38.  Continuous  Data  Histogram  for  Other  Air  Vehicle . 80 

Figure  39.  Cook's  D  Influence  for  Other  Air  Vehicle . 8 1 

Figure  40.  Normality  Histogram  for  Other  Air  Vehicle . 83 

Figure  4 1 .  Autocorrelation  Function  for  Other  Air  Vehicle . 84 


viii 


Page 


Figure  42.  Residual  by  Predicted  Plot  for  Other  Air  Vehicle . 85 

Figure  43.  Plot  of  Residuals  for  Other  Air  Vehicle . 86 

Figure  44.  Continuous  Data  Histogram  for  Flyaway  Cost . 86 

Figure  45.  Cook's  D  Influence  for  Flyaway  Cost . 87 

Figure  46.  Normality  Histogram  for  Flyaway  Cost . 88 

Figure  47.  Autocorrelation  Function  for  Flyaway  Cost . 89 

Figure  48.  Residual  by  Predicted  Plot  for  Flyaway  Cost . 90 

Figure  49.  Plot  of  Residuals  for  Flyaway  Cost . 91 

Figure  50.  Frequency  Chart  of  Flyaway  Bootstrap  Data . 92 

Figure  51.  Frequency  Chart  of  Basic  Airframe  Bootstrap  Data . 93 

Figure  52.  Differenced  Flyaway  vs.  Basic  Airframe  Data . 93 

Figure  53.  Frequency  Chart  of  Other  Air  Vehicle  Bootstrap  Data . 94 

Figure  54.  Differenced  Flyaway  vs.  Other  Air  Vehicle  Data . 95 

Figure  55.  Distributions  from  Macro  vs.  Micro  Comparisons . 96 


IX 


List  of  Tables 


Table  Page 

Table  1.  DoD  Annual  Budget  Authority  (Cohen,  2000:Appendix  B) . 3 

Table  2.  Air  Force  Aircraft  Inventory  (Cohen,  2000:Appendix  D) . 4 

Table  3.  Work  Breakdown  Structure  Levels  (Department  of  Defense  (b), 

1998:A.3) . 11 

Table  4.  Methodology  Applicability  by  Phase  (Long,  2000:7) . 20 

Table  5.  PCEI  Pilot  Program  Site  Results  (Long,  2000:8) . 21 

Table  6.  Independent  Variables  Considered  for  CER  Development . 34 

Table  7.  Portion  of  F-16  Independent  Variables  Data . 43 

Table  8.  ANOVA  Table  for  Basic  Airframe . 47 

Table  9.  Parameter  Estimates  for  Basic  Airframe . 48 

Table  10.  Summary  of  Fit  for  Basic  Airframe . 49 

Table  11.  ANOVA  Table  for  Other  Air  Vehicle . 55 

Table  12.  Summary  of  Fit  and  Parameter  Estimates  for  Other  Air  Vehicle . 57 

Table  13.  Quartiles  for  Other  Air  Vehicle . 59 

Table  14.  ANOVA  Table  for  Flyaway  Cost . 63 

Table  15.  Summary  of  Fit  and  Parameter  Estimates  for  Flyaway  Cost . 64 

Table  16.  Quartiles  for  Flyaway  Cost . 65 

Table  17.  ANOVA  Table  for  Basic  Airframe . 73 

Table  18.  Summary  of  Fit  and  Parameter  Estimates  for  Basic  Airframe . 76 

Table  19.  Quartiles  for  Basic  Airframe . 77 

Table  20.  ANOVA  Table  for  Other  Air  Vehicle . 81 


x 


Page 


Table  2 1 .  Summary  of  Fit  and  Parameter  Estimates  for  Other  Air  Vehicle . 82 

Table  22.  Quartiles  for  Other  Air  Vehicle . 83 

Table  23.  ANOVA  Table  for  Flyaway  Cost . 87 

Table  24.  Summary  of  Fit  and  Parameter  Estimates  for  Flyaway  Cost . 88 

Table  25.  Quartiles  for  Flyaway  Cost . 89 

Table  26.  Practical  Significance  of  WBS  Estimation  Levels . 100 


AFIT/GCA/ENV/03-08 


Abstract 

A  shrinking  workforce,  unstable  budgets,  and  rapidly  changing  objectives  under 
stricter  time  constraints  characterize  today’s  cost  analysis  and  acquisition  environment. 

In  concert  with  this  environment,  cost  analyst  positions  have  rapidly  decreased  as 
demonstrated  by  Aeronautical  Systems  Centers  54%  decline  in  total  authorized  slots  from 
1992  to  2001.  The  question  is  how  to  deal  with  this  ‘more  with  less’  mentality. 

The  purpose  of  this  research  is  to  investigate  and  measure  the  risks  associated 
with  taking  a  macro  versus  micro  approach  to  aircraft  cost  estimation.  By  analyzing  the 
fidelity  of  a  cost  estimate  developed  at  the  flyaway  cost  level  versus  a  cost  estimate 
developed  at  the  individual  components  level,  this  research  provides  guidelines  for 
appropriate  allocation  of  cost  analyst  resources.  This  objective  is  accomplished  by 
looking  at  the  cost  estimation  error  risk  of  recurring  costs  at  level  one  of  the  Work 
Breakdown  Structure  (WBS)  and  at  level  two  of  the  WBS. 

Results  show  that  there  is  a  statistically  significant  difference  between  estimating 
at  the  differing  WBS  levels.  However,  from  a  practical  standpoint,  the  difference  in 
dollar  terms  is  too  small  to  be  considered  significant.  As  a  result,  program  manager 
should  allocate  resources  based  on  other  constraints  such  as  time  allotted  to  complete  the 
estimate  or  required  level  of  visibility  into  the  estimate. 
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A  COMPARATIVE  ANALYSIS  OF  THE  COST  ESTIMATING  ERROR  RISK 


ASSOCIATED  WITH  FLYAWAY  COSTS  VERSUS  INDIVIDUAL  COMPONENTS 

OF  AIRCRAFT 


I.  Introduction 


Background 

A  shrinking  workforce,  unstable  budgets,  and  rapidly  changing  objectives  under 
stricter  time  constraints  characterize  today’s  cost  analysis  and  acquisition  environment. 
The  end  result  of  this  environment  is  that  today’s  cost  community  is  being  asked  to  do 
more  with  less  (Cho,  Jerrel,  and  Landley,  2000: 1-1).  This  is  driving  the  need  for  cost 
analysts  to  increase  their  productivity  or  identify  and  concentrate  on  those  areas  of 
analysis  that  encompass  the  majority  of  estimation  error  risk  in  order  to  meet  the  demand. 

The  Department  of  Defense  (DoD)  civilian  workforce  has  been  downsized  by 
420,000  employees  over  the  past  decade,  a  ten  percent  decrease,  with  an  additional 
70,000  job  cuts  projected  by  the  end  of  fiscal  year  2005  (Garamone  (a),  2000: 10).  While 
this  trend  in  civilian  downsizing  is  consistent  across  the  cost  analysis  career  field, 
military  cost  analyst  slots  are  also  declining.  This  phenomenon  is  taking  place  across  all 
Air  Force  installations,  including  those  at  the  highest  level  (see  Figure  1)  as  addressed  by 
the  Air  Force  Cost  Analysis  Agency  (AFCAA): 
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During  1998-1999,  AFCAA  saw  a  continual  decrease  in  personnel  numbers, 
especially  on  the  military  side.  The  shrinking  numbers  are  a  familiar  occurrence 
across  the  entire  Air  Force  as  the  cost  analysis  field  has  become  absorbed  into  the 
overall  financial  management  field  (Deputy,  1999:53). 


AFCAA  Personnel  Changes 


Number 


Year 


□  Military 
■  Civilian 


Figure  1.  AFCAA  Personnel  Changes  (Deputy,  1999:53) 


These  reductions  in  manpower  have  impacted  operations  level  organizations  such 
as  the  Aeronautical  Systems  Center  (ASC)  cost  analyst  resources  located  at  Wright- 
Patterson  AFB,  OH.  Since  1992,  ASC’s  total  authorized  cost  analyst  slots  have  declined 
by  54%,  from  136  authorizations  to  only  63  in  2001  (Ruffner,  2002).  This  includes  a 
69%  loss  of  military  slots  and  a  44%  drop  in  civilian  authorizations  (Ruffner,  2002). 

While  the  workforce  has  seen  steady  declines  over  the  past  decade,  and  will 
continue  to  see  them  into  the  future,  defense  funding  has  also  shown  instability.  Both  the 
overall  DoD  and  Air  Force  budgets,  including  procurement  funding,  experienced  sharp 
declines  during  the  early  1990’s  (see  Table  1). 
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Table  1.  DoD  Annual  Budget  Authority  (Cohen,  2000:  Appendix  B) 


Base  Year 
(BY)  01$M 

FY 

91 

FY 

92 

FY 

93 

FY 

94 

FY 

95 

FY 

96 

FY 

97 

FY 

98 

FY 

99 

FY 

00 

FY 

01 

DoD  Proc. 
Budget 

84 

72 

59 

48 

47 

45 

45 

47 

52 

55 

60 

DoD  Total 
Budget 

346 

345 

318 

293 

292 

284 

282 

277 

292 

288 

291 

Air  Force 
Proc.  Budget 

29 

27 

24 

19 

17 

18 

15 

16 

19 

19 

21 

Air  Force 
Total  Budget 

112 

101 

94 

87 

84 

83 

80 

82 

85 

85 

86 

The  result  of  these  funding  decreases  has  been  a  slowdown  in  new  aircraft 
acquisitions  starts,  driving  current  aircraft  to  remain  in  service  longer  than  initially 
planned.  Thus,  the  current  aircraft  fleet  is  rapidly  aging,  with  the  average  age  of  Air 
Force  aircraft  at  22  years  (Garamone  (b),  2000).  This  problem  has  caught  the  attention  of 
the  Chief  of  Staff  of  the  Air  Force,  “In  15  years  it  (average  age  of  Air  Force  aircraft  in 
years)  will  be  nearly  30,  even  if  we  execute  every  modernization  program  we  currently 
have  on  the  fiscally  constrained  books”  (Garamone  (b),  2000).  This  aging  aircraft 
problem  has  resulted  in  calls  for  an  increase  in  the  number  of  modernization  programs 
and  funding  in  future  defense  budgets  (Druyan,  2001  :iv).  Future  Presidential  budgets  are 
acting  on  these  modernization  requests  by  increasing  projected  funding.  Therefore,  a 
trend  towards  larger  defense  budgets,  including  procurement,  appears  likely  to 
materialize  over  the  next  several  years  (Department  of  Defense  (d),  2001:  table  6.1). 

Although  the  workforce  has  been  shrinking  and  the  funding  unstable,  there  have 
been  no  significant  declines  in  existing  acquisition  programs  or  inventories  of  aircraft 
these  programs  support  (see  Table  2). 
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Table  2.  Air  Force  Aircraft  Inventory  (Cohen,  2000:Appendix  D) 


Active  & 
Reserve  A/C 

FY94 

FY95 

FY96 

FY97 

FY98 

FY99 

FY00 

FY01 

Inter-theater 

Airlift 

808 

802 

799 

775 

756 

756 

733 

722 

Bombers 

151 

140 

125 

126 

138 

143 

152 

154 

Fighters 

1605 

1512 

1440 

1440 

1440 

1455 

1485 

1455 

Total 

Inventory 

2564 

2454 

2364 

2341 

2334 

2354 

2370 

2331 

The  workload  and  product  demand  from  the  typical  system  program  office  (SPO)  has 

remained  relatively  stable  despite  these  declines  in  manpower  and  funding. 

The  objectives  of  acquisition  programs,  however,  are  quickly  changing.  The 

“world  is  entering  the  ‘era  of  globalization’”  (Garamone  (c),  2001:2)  and  the  threat  to  the 

United  States  is  rapidly  changing  from  nations  engaged  in  a  Cold  War  to  rogue  terrorist 

factions.  This  changing  paradigm  was  clearly  brought  into  focus  after  the  terrorist  attacks 

in  New  York  City  and  Washington  D.C.,  and  the  ensuing  military  operations  in 

Afghanistan.  The  result  of  these  events  is  that  the  Air  Force  is  quickly  moving  to  take 

advantage  of  new  technology  and  develop  new  programs  that  will  be  able  to  “respond 

with  the  lethality  and  precision  to  pop-up  targets  in  seconds  or  minutes,  not  hours,  not 

days”  (Grier,  2002:3).  Although  manpower  levels  have  not  increased,  aggressive  goals 

are  being  implemented  to  field  weapon  systems  at  a  much  more  rapid  rate: 

The  threat  environment  is  now  unstable  and  constantly  changing.  Therefore,  we 
need  to  be  more  flexible  and  responsive  in  meeting  the  needs  of  our  warfighters 
by  fielding  new  systems  in  much  less  time.  Our  goal  is  to  reduce  the  cycle  time 
of  new  programs  by  25  percent  (Cohen  Reports,  2000:  75). 

Problem  Statement 


The  current  aircraft  acquisition  environment  presents  several  challenges  to  the 
cost  analysis  community.  First,  cost  analysts  must  operate  within  the  reality  of  a  smaller 
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workforce,  while  still  accomplishing  their  mission  of  providing  the  best  possible  cost 
analysis  and  estimating  for  their  program.  Second,  cycle  time  reduction  goals  require 
cost  analysts  to  complete  estimates  in  a  compressed  timeframe.  Finally,  in  this 
unpredictable  environment,  cost  analysts  do  not  have  the  luxury  of  knowing  in  advance 
future  estimation  requirements.  Thus,  the  ability  to  accomplish  data  collection  in  support 
of  developing  low-level,  grass-root  estimates  will  be  greatly  reduced. 

Despite  these  changes  in  time,  manpower,  and  predictability,  it  is  extremely 
important  that  weapon  systems  perfonn  at  optimal  operating  capabilities.  Achieving  this 
objective  necessitates  the  highest  quality  level  of  work  from  cost  and  acquisition 
personnel  be  maintained,  “With  budgets  shrinking  and  requirement  steadily  growing, 
DoD  has  logically  focused  on  initiatives  to  increase  efficiency”  (Ciccotello,  Green,  and 
Homyak,  1997:28).  Detennining  methods  to  meet  these  challenges  is  imperative  for  cost 
analysts  in  today’s  environment.  In  order  to  keep  the  quality  of  work  high,  with  less 
personnel,  increasing  programs,  and  dynamic  technology,  analysts  will  be  required  to 
increase  not  only  productivity,  but  also  efficiency.  To  achieve  increased  productivity  and 
efficiency  under  these  conditions,  cost  analysts  must  recognize  where  in  a  new  weapon 
system  the  greatest  estimation  error  risk  resides.  These  high-risk  areas  must  then  be 
where  efforts  are  concentrated  when  developing  an  aircraft  cost  estimate. 

Research  Objectives 

The  purpose  of  this  research  is  to  investigate  and  measure  the  risks  associated 
with  taking  a  macro  versus  micro  approach  to  aircraft  cost  estimation.  By  analyzing  the 
fidelity  of  a  cost  estimate  developed  at  the  flyaway  cost  level  versus  a  cost  estimate 
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developed  at  the  individual  components  level,  this  research  will  provide  guidelines  for 

appropriate  allocation  of  cost  analyst  resources  in  today’s  constrained  environment. 

Flyaway  costs  for  aircraft  are  defined  as  follows: 

It  relates  to  production  cost  and  includes  the  prime  mission  equipment  (basic 
structure,  propulsion,  electronics),  systems  engineering,  program  management, 
and  allowances  for  engineering  changes  and  warranties.  Flyaway  costs  include 
(all)  recurring  . . .  production  costs  (contractor  and  Government  furnished 
equipment)  that  are  incurred  in  the  manufacture  of  a  usable  end-item  (AFSC  Cost 
Estimating  Handbook,  1986:217). 

In  particular,  two  categories  of  aircraft  will  be  considered:  fighters  and  inter¬ 
theater  airlift.  Inter-theater  airlift  are  those  aircraft  used  for  supply  and  transportation. 

The  following  questions  regarding  each  of  these  categories  will  be  addressed: 

1 .  Which  aircraft  components  have  the  most  cost  estimation  error  risk,  and  what  is 
that  risk? 

2.  What  is  the  cost  estimation  error  risk  associated  with  estimating  at  the  flyaway 
cost  level? 

3.  Is  there  a  statistically  significant  difference  in  estimating  at  the  component  level 
versus  the  flyaway  level? 

4.  Given  a  constrained  resource  environment,  where  should  a  cost  analyst  focus  their 
attention  when  developing  an  aircraft  cost  estimate? 

Methodology 

These  research  objectives  will  be  accomplished  by  analyzing  historical  data  at 
both  the  aircraft  component  cost  and  flyaway  cost  level.  Aircraft  under  review  will  be 
separated  into  two  categories  to  analyze:  fighters  and  inter-theater  airlift.  Cost  data  for 
these  categories  of  aircraft  will  be  acquired  from  the  Cost  Estimating  System  Volume  2, 


6 


Aircraft  Cost  Handbook,  Book  1 :  Aircraft,  Nov  87,  which  was  prepared  for  the  AFCAA 
by  Delta  Research  Corporation. 

Individual  component  costs  for  each  category  of  aircraft  will  be  subdivided  into 
two  parts,  basic  airframe  costs  and  other  air  vehicle  costs.  Basic  airframe  costs  will 
include  engineering,  tooling,  manufacturing,  quality  control  and  other  airframe  costs. 
Other  airframe  costs  are  comprised  of  purchased  equipment,  material  overhead, 
subcontractor  costs,  other  costs  not  shown  elsewhere,  and/or  General  and  Administrative 
(G&A)  and  fees  associated  with  outside  production  and  services.  Other  air  vehicle  costs 
will  include  system  test  and  evaluation,  program  management,  and  other  (for  example 
avionics  subsystems).  Cost  estimating  relationships  (CERs)  at  the  individual  aircraft 
components  level  and  flyaway  cost  level  will  then  be  developed  using  linear  regression 
techniques. 

A  statistical  analysis  of  these  CERs  will  be  accomplished  to  determine  the  standard 
error  associated  with  each  CER  through  the  use  of  Monte  Carlo  simulation  techniques. 
The  component  level  and  flyaway  level  error  risk  will  be  compared  for  statistically 
significant  differences.  These  results  will  provide  guidelines  for  areas  of  emphasis  when 
developing  an  aircraft  cost  estimate  when  time  and  resources  to  conduct  a  grass-roots 
estimate  are  constrained. 

Scope  and  Limitations 

The  scope  of  this  study  is  limited  to  the  allocation  of  cost  analyst  resources  for 
aircraft  programs.  Specifically,  this  study  is  tailored  for  applicability  to  program  offices 
located  at  ASC.  The  methodology  used  for  this  research  may  be  applicable  to  other 
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human  resource  functional  categories  or  to  other  categories  of  weapon  systems. 

However,  any  such  extrapolation  of  this  methodology  will  first  require  a  full 
investigation  of  the  risks  involved  in  any  such  application. 

Thesis  Overview 

Chapter  two  addresses  the  fundamental  concepts  and  techniques  of  aircraft  cost 
estimation  and  the  previous  research  conducted  in  this  area.  Chapter  three  explains  the 
methodology  behind  this  research.  Chapter  four  applies  the  methodology  of  chapter  three 
to  analyze  the  data.  Chapter  five  clarifies  the  results  and  conclusions  that  can  be  derived 
from  chapter  four’s  analysis.  Future  research  is  also  outlined  in  this  chapter. 
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II.  Literature  Review 


Chapter  Overview 

This  chapter  focuses  on  the  fundamental  components  and  techniques  used  to 
develop  an  aircraft  cost  estimate.  First,  a  discussion  of  the  basic  building  block  for  any 
cost  estimate,  the  Work  Breakdown  Structure  (WBS),  is  examined.  Next,  the  role  of 
CERs  in  aircraft  estimation  is  explored  to  understand  why  and  how  they  are  used.  Then, 
an  explanation  of  aircraft  cost  estimation  techniques,  specifically  the  parametric  and  grass 
roots  methods,  are  covered.  Finally,  an  overview  of  past  research  that  has  been 
accomplished  on  comparisons  of  macro  and  micro  aircraft  estimation  techniques  is 
investigated. 

Work  Breakdown  Structure 

Due  to  its  versatility,  the  WBS  is  a  useful  tool  for  both  program  managers  and 

cost  analysts  in  defining  the  scope  of  a  project.  The  WBS  is  formally  defined  as: 

A  deliverable-oriented  grouping  of  project  elements  that  organizes  and  defines  the 
total  scope  of  the  project:  work  not  in  the  WBS  is  outside  the  scope  of  the  project. 
As  with  the  scope  statement,  the  WBS  is  often  used  to  develop  or  confirm  a 
common  understanding  of  project  scope.  Each  descending  level  represents  an 
increasingly  detailed  description  of  project  elements  (PMI  Standards  Committee, 
1996:54). 

The  overarching  objective  of  the  WBS  is  to  integrate  a  project  into  a  singularly  unique 
and  cohesive  effort  (Flemming  and  Koppelman,  2000:49). 

The  WBS  is  a  basic  building  block  of  all  Major  Defense  Acquisition  Programs 
(MDAPs).  As  such,  DoD  Regulation  5000. 2-R  mandates  that,  “A  program  work 
breakdown  structure  (WBS)  shall  be  established  that  provides  a  framework  for  program 


9 


and  technical  planning,  cost  estimating,  resource  allocation,  performance  measurement, 
and  status  reporting”  (Department  of  Defense  (a),  1996:  Part  4).  In  addition  to 
developing  a  WBS,  every  program  office  is  required  to  tailor  their  WBS  using  the 
guidelines  set  forth  in  Military  Handbook-881  (Department  of  Defense  (a),  1996:  Part  4). 

The  benefits  associated  with  using  a  WBS  include: 

•  Separates  a  defense  materiel  item  into  its  component  parts,  making  the 
relationships  of  the  parts  clear  and  the  relationships  of  the  tasks  to  be 
completed-to  each  other  and  to  the  end  product-clear. 

•  Significantly  affects  planning  and  the  assignment  of  management  and 
technical  responsibilities. 

•  Assists  in  tracking  the  status  of  engineering  efforts,  resource  allocations, 
cost  estimates,  expenditures,  and  cost  and  technical  performance. 

•  Helps  ensure  that  contractors  are  not  unnecessarily  constrained  in  meeting 
item  requirements  (Department  of  Defense  (b),  1998:9). 

The  WBS  can  be  broken  down  into  as  many  levels  as  required.  However,  a 
breakout  of  the  top  three  levels  satisfies  all  regulatory  requirements  (Department  of 
Defense  (b),  1998: 1 1).  This  research  will  focus  its  comparisons  between  level  one  and 
level  two  of  the  WBS  to  facilitate  the  macro  versus  micro  properties.  Level  two  is 
selected  as  the  micro  level  because  of  data  availability  and  the  fact  that,  “Level  two  of 
any  WBS  is  the  most  critical,  because  at  level  two  the  project  manager  will  indicate  the 
approach  planned  to  manage  the  project”  (Fleming  and  Koppehnan,  2000:54). 

Level  one  of  the  WBS  is  the  entire  defense  material  item,  represented  in  this 
research  by  a  complete  aircraft  system.  Level  two  of  the  WBS  is  the  major  elements  that 
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comprise  the  aircraft  system.  Level  two  includes  equipment  specific  elements  and 
common  elements  found  in  all  major  weapons  systems.  These  common  elements  include 
systems  engineering  and  program  management,  training,  data,  system  test  and  evaluation, 
etc.  The  guidelines  for  the  WBS  structure  of  an  aircraft  system  come  from  Military 
Handbook  Standard  881  (MIL-HDBK-881),  as  shown  in  Table  3. 


Table  3.  Work  Breakdown  Structure  Levels  (Department  of  Defense  (b),  1998:  A.3) 


Level  1 

Level  2 

Level  3 

Aircraft  System 

Air  Vehicle  (AV) 

Airframe 

Propulsion 

AV  Applications  Software 

AV  System  Software 

C  ommunic  ations/Identific  ation 
Navigation/Guidance 

Central  Computer 

Fire  Control 

Data  Display  and  Controls 
Survivability 

Reconnaissance 

Automatic  Flight  Control 

Central  Integrated  Checkout 
Antisubmarine  Warfare 

Armament 

Weapons  Delivery 

Auxiliary  Equipment 

Sys  Engineering/Program 
Management 

System  Test  and  Evaluation 

Development  Test  and  Evaluation 
Operational  Test  and  Evaluation 
Mock-ups 

Test  and  Evaluation  Support 

Test  Facilities 

Training 

11 


Level  1 

Level  2 

Level  3 

Equipment 

Services,  Facilities 

Data 

Technical  Publications 

Engineering  Data 

Management  Data 

Support  Data 

Data  Depository 

Peculiar  Support  Equipment 

Test  and  Measurement  Equipment 
Support  and  Handling  Equipment 

Common  Support 

Equipment 

Test  and  Measurement  Equipment 
Support  and  Handling  Equipment 

Operational/Site  Activation 

System  Assembly,  Installation 
and 

Checkout  on  Site 

Contractor  Technical  Support 

Site  Construction 

Site/Ship/Vehicle  Conversion 

Industrial  Facilities 

Construction/Conversion/Expan. 
Equipment  Acquisition  or 
Modernization 

Maintenance  (Industrial 

Facilities) 

Initial  Spares  and  Repair  Parts 
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WBS  Terminology  Clarification 

While  the  suggested  WBS  structure  is  being  followed  for  data  collection  and 
analysis  purposes,  there  are  some  terminology  differences  between  MIL-HDBK-881  and 
the  subsequent  language  used  to  describe  the  data  collected.  Specifically,  at  WBS  level 
one,  the  term  Flyaway  Cost  is  substituted  for  Aircraft  System.  This  change  is  made 
because  program  office  costs  and  costs  not  directly  related  to  the  contractor  are  not  being 
considered.  At  WBS  level  two,  the  tenn  Basic  Airframe  is  substituted  for  Air  Vehicle. 
Also,  the  common  elements  of  System  Engineering/Program  Management,  System  Test 
and  Evaluation,  Data,  and  Training  are  reclassified  into  a  single  category  called  Other  Air 
Vehicle.  The  form  of  the  available  data  for  collection  drives  these  changes. 

Cost  Estimating  Relationships 

The  CER  is  one  of  the  fundamental  techniques  used  to  estimate  aircraft  cost.  A 
CER  is  formally  defined  as  a  “technique  used  to  estimate  a  particular  cost  or  price  by 
using  an  established  relationship  with  an  independent  variable”  (Office  of  the  Deputy 
Director  of  Defense  Procurement  for  Cost,  Pricing,  and  Finance,  2000: 1).  The  dependent 
variable  is  the  item  of  interest  that  the  CER  will  estimate,  (i.e.  airframe  cost).  The 
independent  variables  are  comprised  of  a  multitude  of  explanatory  variables.  The  CER  is 
a  mathematical  relationship  that  predicts  the  dependent  variable  as  a  function  of  the 
independent  variables.  This  relationship  is  typically  developed  using  a  historical  dataset 
of  variables  and  applying  a  statistical  technique,  usually  regression,  to  find  the  parameter 
estimates  of  the  independent  variables  (Younassi,  Kennedy,  and  Graser,  2001:83). 
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The  selection  of  independent  variables  is  extremely  important.  To  ensure  an 
accurate  and  meaningful  CER  is  developed,  the  independent  variables  must  be  identified 
as  cost  drivers  for  the  dependent  variable.  “Cost  drivers  are  those  characteristics  of  a 
product  or  item  that  have  a  major  effect  on  the  product  or  item  cost”  (Long,  2000:2). 
Typically,  performance  parameters  are  the  most  useful  and  accurate  independent 
variables  (AFSC  Cost  Estimating  Handbook,  1986:6. 1),  however  physical  and  technical 
variables  are  common  in  CERs.  Identification  of  cost  drivers  to  include  in  the  CER 
depends  on  the  type  of  CER  being  developed.  Depending  on  the  life  cycle  phase  of  the 
program,  CERs  can  be  categorized  into  three  types:  Research  and  Development, 
Production,  or  Operating  and  Support  (AFSC  Cost  Estimating  Handbook,  1986:6.3.1). 
This  research  will  focus  on  aircraft  production  CERs.  Past  research  has  identified 
conventional  cost  drivers  for  aircraft  CERs  to  include  empty  weight,  speed,  useful  load, 
wing  area,  power,  landing  speed,  and  production  quantity  (Office  of  the  Deputy  Director 
of  Defense  Procurement  for  Cost,  Pricing,  and  Finance,  2000:3). 

Historical  data  is  typically  used  to  populate  the  statistical  models  that  generate  the 
parameter  estimates  for  the  independent  variables.  “Collecting  data  is  usually  the  most 
difficult  and  time-consuming  element  of  CER  development”  (Office  of  the  Deputy 
Director  of  Defense  Procurement  for  Cost,  Pricing,  and  Finance,  2000:2).  The  advantage 
of  using  historical  data  is  that  it  has  an  objective  and  proven  level  of  cost  perfonnance 
(Institute  for  Defense  Analysis,  2000: 1).  However,  special  care  must  be  taken  to  ensure 
that  the  data  collected  is  free  of  errors  or  other  influential  factors  that  may  skew  a  CER 
under  development.  Another  limitation  of  using  historical  data  in  CER  development  is 
that  it  focuses  on  the  past,  while  the  purpose  is  to  predict  the  future  (Institute  for  Defense 
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Analysis,  2000:1).  Fortunately,  functional  relationships  for  items  such  as  aircraft 
structure  have  remained  relatively  stable  for  the  past  several  decades  (Institute  for 
Defense  Analysis,  2000:1). 

The  most  common  statistical  technique  used  to  develop  aircraft  CERs  is  the 
multiple  least  squares  regression  method.  To  ensure  the  resultant  CER  from  the 
regression  model  is  valid,  several  assumptions  and  tests  must  be  conducted.  First,  the 
assumptions  of  independence,  nonnality,  and  constant  variance  must  be  satisfied 
(Devore,  2000:547).  A  Lag  1  correlation  test,  such  as  the  Durbin-Watson  test,  can 
analyze  the  assumption  of  independence  (Neter,  Kutner,  Nachtsheim,  and  Wasserman, 
1996:504).  At  an  a  =  0.05  level,  the  Durbin-Watson  test  must  return  a  p-value  of  greater 
than  0.05  to  demonstrate  independence.  If  a  lower  p-value  is  realized,  the  data  is  not 
useful  and  the  CER  should  be  considered  invalid.  The  Shapiro-Wilk  test  is  an  objective 
measure  to  determine  if  the  assumption  of  normality  is  met  (Neter,  Kutner,  Nachtsheim, 
and  Wasserman,  1996: 1 1 1).  Conducted  at  an  a  =  0.05  level,  the  assumption  is  validated 
if  the  test  returns  a  p-value  greater  than  0.05.  Similarly,  the  assumption  of  constant 
variance  can  be  confirmed  using  the  Breusch-Pagan  test  if  the  p-value  is  greater  than  the 
a  =  0.05  threshold  (Neter,  Kutner,  Nachtsheim,  and  Wassennan,  1996:115).  While  the 
assumption  of  independence  is  non-negotiable  in  achieving  a  valid  regression  model,  the 
assumptions  of  normality  and  constant  variance  can  be  violated  if  it  can  be  shown  that  the 
violation  is  a  “reasonable  violation”  (Osborne  and  Waters,  2002: 1).  Detennining  if  a 
violation  of  an  assumption  is  “reasonable”  is  the  responsibility  of  the  modeler  based  on 
the  known  facts  of  the  individual  case.  After  the  assumptions  have  been  met,  it  is 
imperative  that  the  regression  model  be  tested  for  validity.  This  is  accomplished  by 
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populating  the  CER  with  a  randomly  selected  portion  of  the  raw  data.  This  randomly 
selected  portion  of  raw  data  is  set  aside  prior  to  building  the  regression  model.  The 
validity  of  the  model  can  then  be  determined  through  analysis  of  the  Mean  Square  Error 
(MSE)  resulting  from  this  data  and  the  model.  The  complete  CER  development  process 
is  shown  in  Figure  2. 
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TO  COST  MODELS 


Figure  2.  CER  Development  Process  (Department  of  Defense  (c),  1995:111-10) 

CERs  are  prevalent  in  many  different  cost  estimation  techniques.  They  are  the 
cornerstones  of  the  parametric  estimation  technique  developed  by  the  RAND  Corporation 
in  the  1950’s  to  predict  the  cost  of  aircraft  (Nhysoft  Inc,  2002).  As  such,  they  are  now 
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the  primary  component  underlying  most  commonly  used  parametric  software  estimating 
suites  (Air  Force  Material  Command,  2001).  The  versatility  of  CERs  can  be  shown  by 
their  cross  utilization  among  other  estimation  techniques.  The  grass  roots  technique  uses 
CERs  to  develop  detailed  labor  and  material  estimates  which  are  then  summed  as 
components  of  the  total  estimate  (AFSC  Cost  Estimating  Handbook,  1986:9.4.1). 

Because  CERs  are  versatile  and  widespread,  they  can  be  found  in  virtually  every  cost 
analyst’s  toolbox. 

There  are  several  characteristics  that  make  CERs  desirable  across  these  cost 
estimation  techniques.  First,  they  are  able  to  “provide  quick  estimates  without  a  great 
deal  of  detailed  information”  (Long,  2000:2).  This  is  important,  as  a  CER  can  be  used 
early  in  a  program’s  life,  before  any  actual  data  is  available,  to  forecast  and  plan  for 
future  budgets.  Second,  because  CERs  are  based  on  historical  data,  they  incorporate  the 
impacts  of  system  growth,  schedule  changes,  and  engineering  changes  (AFSC  Cost 
Estimating  Handbook,  1986:3.4. 1).  These  changes  are  a  fact  of  virtually  every  DoD 
program.  Because  these  items  are  part  of  the  historical  data,  the  CER  is  able  to  give  a 
more  realistic  picture  of  the  future.  Most  importantly,  CERs  have  proven  to  be  good 
predictors,  which  is  the  goal  of  any  cost  estimation  technique. 

Aircraft  Estimation  Techniques 

A  variety  of  estimation  techniques  are  available  to  the  cost  analyst  for  developing 
aircraft  cost  estimates.  The  two  ends  of  the  estimation  technique  spectrum  are  parametric 
estimation  and  grass  roots  estimation.  The  parametric  estimation  technique  can  be 
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considered  a  macro  approach  to  cost  estimation,  while  the  grass  roots  approach  is 
consistent  with  a  micro  approach  to  cost  estimation. 

Parametric  Estimation 

In  today’s  acquisition  environment  of  doing  more  with  less,  parametric  estimating 

has  become  a  common  tool  for  the  cost  analyst.  Parametric  estimation  can  be  defined  as: 

A  technique  employing  one  or  more  CERs  and  associated  mathematical 
relationships  and  logic.  The  technique  is  used  to  measure  and/or  estimate  the  cost 
associated  with  the  development,  manufacture,  or  modification  of  a  specified  end 
item.  The  measurement  is  based  on  the  technical,  physical,  or  other  end  item 
characteristics  (Department  of  Defense  (c),  1995:16). 

The  CERs  developed  to  populate  the  parametric  cost  model  are  typically  derived  through 

non-experimental  regression  techniques  (Robbins  and  Daneman,  1999:24). 

The  parametric  cost  model  represents  the  macro  approach  to  estimation  for 

several  reasons.  First,  the  focus  is  on  high-level  cost  drivers  and  high-level  data  from 

which  trends  can  be  extracted  (Nhysoft  Inc,  2002).  Second,  the  parametric  method  is 

often  used  early  in  the  acquisition  cycle  when  program  and  technical  definition  is  limited 

(AFSC  Cost  Estimating  Handbook,  1986:3.4. 1).  At  this  point  in  the  life  cycle,  the  details 

to  develop  a  comprehensive  estimate  are  scarce,  so  the  parametric  estimate  is  a  more 

useful  estimation  tool.  Finally,  capturing  total  program  costs  can  be  accomplished  with  a 

single  parametric  model  (AFSC  Cost  Estimating  Handbook,  1986:3.4.1).  This  one-size 

fit  all  approach  can  be  characterized  as  a  macro  technique. 

The  stochastic  nature  of  cost  estimates  is  changing  the  goal  of  cost  estimates  to 

that  of  cost  realism.  This  paradigm  shift  from  desiring  an  exact  estimate  to  one  in  which 

the  foundation  of  the  cost  estimate  is  more  important  lends  credibility  to  more 

widespread  use  of  parametric  cost  models  (Department  of  Defense  (c),  1995:17).  This  is 
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because  parametric  estimates  are  built  on  the  foundation  of  CERs  that  are  typically 
derived  through  regression  techniques  that  attempt  to  minimize  the  sum  of  square  errors. 
The  presence  of  an  error  factor  in  the  estimate,  while  minimized,  is  recognized  as  a 
fundamental  component  of  parametric  estimates.  Therefore,  the  shift  in  emphasis  to  cost 
realism  facilitates  the  expansion  of  parametric  techniques  from  its  origins  as  a  technique 
primarily  used  in  the  early  stages  of  an  acquisition  program,  to  one  used  throughout  the 
life  cycle  phases  (see  Table  4). 

Table  4.  Methodology  Applicability  by  Phase  (Long,  2000:7) 


Life  Cycle  Phases 

Parametric 

Bottom  Up 

Conceptual 

X 

Development 

X 

X 

Production 

X 

X 

O&S 

X 

Parametric  estimation  has  become  the  preferred  method  for  many  organizations 
within  the  DoD,  including  the  Defense  Contract  Audit  Agency  (DCAA).  “The  DC  A  A 
policy  is  clear  -  parametric  cost  estimating  is  a  preferred  approach  based  on  proven 
statistical  concepts  and  techniques”  (Long,  2000:8).  These  policy  decisions  are  a  result 
of  the  multitude  of  benefits  derived  from  using  parametric  techniques.  These  benefits 
include  the  ability  to  complete  a  parametric  cost  estimate  in  less  time  than  a  traditional 
grass-roots  estimate.  Additionally,  parametric  techniques  are  more  user  friendly  due  to 
the  inherent  flexibility  of  the  cost  model  (Department  of  Defense  (c).  Intro).  Estimate 
changes  can  be  made  simply  by  changing  parameters.  This  flexibility  in  the  parametric 
model  is  most  useful  for  sensitivity  analysis,  which  is  common  in  aircraft  SPOs  due  to 
typical  programmatic  changes  such  as  quantity  changes,  scope  changes,  and  budget  cuts 
Additionally,  parametric  estimates  have  proven  to  be  very  accurate,  even  for  complex 
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systems  (Nhysoft  Inc,  2002).  These  benefits  of  parametric  cost  models  are  most  clearly 
seen  by  examining  the  results  of  a  pilot  study  conducted  on  contractor  proposals 
submitted  to  the  government  by  13  Reinvention  Laboratory  sites  in  1995  (Long,  2000:8). 
This  study  was  a  result  of  the  Parametric  Cost  Estimating  Initiative  (PCEI)  to  examine 
the  viability  of  expanding  the  use  of  parametric  models.  As  shown  in  Table  5,  the 
parametric  models  proved  to  be  better  than  or  equal  to  their  traditional  counterparts  at 
aircraft  cost  estimation  when  tested  with  various  contractors. 


Table  5.  PCEI  Pilot  Program  Site  Results  (Long,  2000:8) 


Site 

Faster 

Cheaper 

Accuracy 

Raytheon 

(E-Systems) 

25% 

33% 

More  Accurate 

Northrup 

Grumman 

ESSD 

33% 

33% 

Equal  or  Better 

Lockheed  Martin 
Astronautics 

25% 

25% 

More  Accurate 

Boeing  SSD 

40% 

35% 

More  Accurate 

Due  to  the  benefits  of  accuracy,  time  efficiencies,  and  cost  savings,  many 
commercial  firms  have  adopted  parametric  cost  models  (Department  of  Defense  (c), 
Background).  DoD  benefits  from  this  commercial  parametric  methodology  adoption 
primarily  in  the  fonn  of  lower  proposal  costs  and  easier  auditability  of  proposals. 

Another  form  of  parametric  estimation  is  the  factor  or  ratio.  These  parametric 
techniques  are  often  applied  to  the  common  elements  found  at  WBS  level  two,  such  as 
training,  data  or  program  management.  For  example,  the  data  cost  for  a  new  aircraft  may 
be  estimated  as  a  factor  of  5%  of  Prime  Mission  Product  (PMP)  based  on  analogous 
programs.  The  primary  benefit  to  using  a  factor  is  that  a  large  part  of  an  estimate  can  be 
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captured  with  limited  historical  data  (Long,  2000:4).  Ratios  are  used  in  a  similar  manner 
as  factors,  and  usually  relate  items  such  as  recurring  to  non-recurring  costs. 

There  are  some  limitations  to  parametric  estimation  that  must  be  considered.  As  a 
result  of  the  cost  typically  being  captured  at  a  high  level,  limited  visibility  into  specific 
areas  is  not  provided.  This  becomes  an  issue  when  subtle  changes  in  design  or 
manufacturing  techniques  occur  in  low  level  areas  and  is  not  reflected  in  the  estimate 
(Long,  2000:5).  In  addition,  the  individual  components  of  an  estimate  may  not  be 
separable,  causing  problems  when  addition  analysis  on  individual  elements  is  needed. 

Grass  Roots  Estimation 

The  grass  roots  technique  for  cost  estimation  is  synonymous  with  the  phrases 
“detailed,”  “bottom-up,”  and  “engineering  build-up”  (AFSC  Cost  Estimating  Handbook, 
1986:3.4.3).  As  implied,  the  underlying  crux  of  a  grass  roots  estimate  is  to  start  at  the 
lowest  level  of  the  WBS,  estimate  the  components,  and  sum  their  parts.  For  this  reason, 
the  grass  roots  estimation  technique  is  categorized  as  a  micro  approach  to  cost  estimation. 

Grass  roots  techniques  are  traditionally  applied  during  the  production  phase  of  a 
weapons  system  for  several  reasons  (AFSC  Cost  Estimating  Handbook,  1986:3.4.3). 

First,  in  order  to  develop  a  high  fidelity  estimate,  the  preparatory  process  requires  a 
highly  detailed  examination  of  the  system  and  program  (AFSC  Cost  Estimating 
Handbook,  1986:3.4.3).  This  process  can  exhaust  a  program’s  valuable  resources. 
Second,  actual  data  from  the  program  is  helpful  to  develop  the  estimate.  This  data  is  not 
available  until  the  production  phase.  Additionally,  it  is  important  that  the  configuration 
of  the  system  is  stable  and  the  test  results  are  available,  as  the  estimate  is  based  on  past 
historical  data  about  the  actual  system  (AFSC  Cost  Estimating  Handbook,  1986:3.4.3). 
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Elements  such  as  configuration  of  the  system  and  test  results  can  dramatically  change  the 
course  of  a  program,  and  stabilization  of  these  elements  does  not  occur  until  the 
production  phase.  While  the  grass  roots  technique  is  typically  used  during  production,  it 
is  important  to  note  that  it  can  also  be  used  to  predict  future  system  costs  during  other 
phases  of  the  life  cycle  with  the  help  of  CERs.  These  CERs  are  used  to  develop  detailed 
labor  and  material  estimates  that  are  then  summed  to  arrive  at  the  cost  estimate  (AFSC 
Cost  Estimating  Handbook,  1986:9.4.1). 

The  application  of  the  grass  roots  technique  normally  requires  detailed  historical 
data.  Often,  the  system  is  broken  down  and  estimated  at  the  various  labor  categories, 
materials,  and  subcontract  components  (AFSC  Cost  Estimating  Handbook,  1986:9.4.1). 
The  summation  of  these  elements  comprises  the  cost  estimate.  There  are  several  benefits 
to  this  approach.  First,  because  there  is  an  extensive  detailed  breakout  of  an  individual 
WBS  element,  it  is  easier  to  scrutinize  and  analyze  deviations  from  the  estimate. 
Additionally,  unlike  the  parametric  technique,  there  is  little  need  for  engineering  support 
when  developing  a  grass  roots  estimate  (AFSC  Cost  Estimating  Handbook,  1986:3.4.3). 
This  is  due  in  large  part  to  the  availability  of  actual  system  data. 

It  is  important  to  recognize  limitations  that  exist  with  the  grass  roots  technique. 
More  time  is  required  to  derive  a  grass  roots  estimate  than  any  other  cost  estimating 
technique  (AFSC  Cost  Estimating  Handbook,  1986:3.4.3).  Also,  actual  data  is  needed 
which  may  or  may  not  be  available.  Even  if  data  is  available,  it  is  often  costly  to 
purchase  it  from  the  contractor.  A  final  limitation  addresses  the  risk  inherent  in  an 
underlying  assumption  of  a  grass  roots  estimate;  that  future  costs  can  be  predicted  from 
historical  costs,  which  may  not  necessarily  be  true  (AFSC  Cost  Estimating  Handbook, 
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1986:3.4.3).  Thus,  the  grass  roots  risk  is  fundamentally  different  from  a  parametric 
estimate,  which  relies  on  causal  relationships  to  achieve  an  estimate.  Despite  these 
limitations,  the  grass  roots  technique  is  widely  used  and  accepted  in  cost  estimation. 

Previous  Research 

This  research  is  the  first  of  its  kind  to  explore  a  statistical  comparison  of  micro 
versus  macro  cost  estimating  techniques.  A  critical  component  for  this  comparative 
analysis  is  the  development  of  CERs  for  level  one  and  level  two  WBS  elements.  The 
RAND  Corporation  is  a  leading  organization  in  analyzing  and  hypothesizing  aircraft 
CERs  (Seibel,  2002).  RAND  studies  on  estimating  aircraft  airframe  costs  date  back  to 
the  1960s.  Several  components  of  these  studies  are  relevant  to  this  research  effort.  For 
example,  while  analyzing  airframe  components  for  a  study,  CERs  were  developed  at  the 
lowest  level  and  compared  to  the  aggregate  level.  In  addition,  RAND  has  examined  the 
benefits  and  detriments  to  segregating  CERs  by  aircraft  categories.  RAND  has  also 
completed  extensive  research  in  identifying  those  explanatory  variables  that  are  of  most 
significance  when  developing  regression  models  for  aircraft  airframes.  This  research  will 
examine  elements  of  these  studies,  to  include  the  segregation  of  aircraft  by  categories, 
identifying  explanatory  variables  to  derive  CERs,  and  analyzing  the  validity  of  micro 
versus  macro  cost  estimation  techniques. 

Large,  Campbell,  and  Cates:  1976 

Parametric  Equations  for  Estimating  Aircraft  Airframe  Costs  provided  an  update 
to  earlier  research  conducted  by  Levenson  and  Barro  (1966)  of  the  RAND  Corporation. 
Large,  Campbell,  and  Cates’  (1976)  goal  was  to  derive  equations  through  multiple 
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regression  techniques  for  estimating  acquisition  costs  of  aircraft  airframes  to  use  in  long 
tenn  planning.  The  scope  of  the  study  encompassed  25  aircraft,  both  development  and 
procurement  phases,  with  first  flight  occurring  between  1953  and  1970. 

A  multitude  of  aspects  relating  to  aircraft  airframe  cost  estimates  was  explored  in 
this  study.  First,  prior  research  in  this  area  had  essentially  found  only  two  main  cost 
drivers:  airframe  unit  weight  and  max  speed.  The  research  implemented  a  three-pronged 
criterion  to  find  other  cost  driver  variables.  For  inclusion  in  the  model,  a  variable  needed 
to  provide  consistently  accurate  estimates,  be  logically  related  to  cost,  and  be  easily 
determined  prior  to  actual  design  and  development.  There  were  no  new  variables  found 
that  met  this  criterion.  A  second  major  contribution  of  this  study  involved  the 
exploration  into  developing  separate  CERs  for  different  categories  of  aircraft.  They 
concluded  that  while  it  was  intuitively  appealing  to  make  such  distinctions,  the  sample 
size  of  aircraft  available  was  too  small  to  be  representative,  with  the  exception  of  fighters. 
Thus,  unless  the  new  aircraft  being  developed  was  considered  very  homogeneous  to  the 
segregated  sample,  it  was  better  to  have  a  larger  group  of  diverse  aircraft  as  the  data 
sample. 

Another  new  aspect  investigated  by  this  study  was  the  difference  in  estimating 
from  equations  developed  for  total  cost  versus  the  individual  elements  of  an  aircraft 
airframe  (i.e.  manufacturing,  tooling,  etc).  Minimal  difference  was  discovered  in  the 
errors  associated  with  the  two  approaches.  Additionally,  it  was  found  that  segregation 
based  on  aircraft  type  for  the  total  cost  equations  was  actually  better,  even  with  a  small 
sample  size,  than  developing  composite  equations  with  all  aircraft  categories  represented. 
Despite  these  findings,  Large,  Campbell  and  Cates  concluded: 
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The  total-cost  equation  has  a  serious  deficiency.  The  equation  is  based  on  the 
composite  rates  above  and  cannot  be  adjusted  to  any  other  set  of  rates.  One  could 
assume  (erroneously)  that  material  costs,  engineering  labor,  tooling  labor,  etc., 
increase  at  the  same  annual  rate  and  in  that  manner  adjust  for  inflation.  But  the 
basic  problem  is  that  the  composite  rates  . . .  may  not  approximate  those  expected 
to  be  in  effect  for  any  specific  program  (Large,  Campbell,  and  Cates,  1976:42). 

Summarizing  their  results,  this  study  developed  estimating  equations  that  were 

similar  to  previous  work  in  this  field,  as  the  primary  explanatory  variables  remain  weight 

and  speed.  Neither  the  total  cost,  nor  individual  element  equations,  produced  estimates 

that  were  consistently  within  20%  of  actual  costs.  Therefore,  the  study  recommended 

further  research  be  conducted  to  examine  non-deterministic  factors  such  as  contractor 

experience,  economic  conditions,  schedule,  etc  as  possible  explanatory  variables  for  this 

inconsistency  in  estimation. 

Hess  and  Romanoff:  1987 

Aircraft  Airframe  Cost  Estimating  Relationships  was  the  third  RAND  study 
examining  this  and  provided  an  update  to  the  1976  study  complied  by  Large,  Campbell, 
and  Cates.  Hess  and  Romanoffs  (1987)  primary  objective  of  the  study  was  to  update 
CERs  for  aircraft  airframes  through  multiple  least  squares  regression  techniques.  This 
update  was  driven  by  the  addition  of  new  aircraft  platfonns  such  as  the  F-15,  F-16,  F-18, 
and  A- 10,  bringing  the  total  number  of  aircraft  analyzed  to  34.  CERs  were  developed  for 
the  individual  components  of  the  airframe  (i.e.  manufacturing,  tooling,  etc.)  and  at  the 
total  airframe  level.  Comparisons  of  the  fidelity  of  the  equations  at  the  varying  levels 
found  virtual  equivalence  in  accuracy. 

Three  sub-objectives  of  the  study  were  also  considered.  First,  the  effect  of 
dividing  the  34  aircraft  into  subcategories  of  homogeneous  data  sets  to  determine  whether 
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better  CERs  resulted  was  examined.  The  categories  used  by  RAND  included  fighters, 
bombers/transport,  and  attack  aircraft.  The  results  of  these  CERs  were  then  compared  to 
the  CERs  developed  at  the  aggregate  aircraft  level.  Hess  and  Romanoff  were  only  able  to 
develop  acceptable  estimating  relationships  for  the  fighter  aircraft  category.  The  inability 
to  develop  CERs  for  the  other  two  categories  was  attributed  to  three  items:  “a  lack  of 
variation  in  perfonnance  variables,  the  heterogeneity  of  the  samples,  and  the  distribution 
of  aircraft  with  respect  to  size”  (Hess  and  Romanoff,  1987:57).  Comparisons  of  these 
CERs  to  the  composite  aircraft  CERs  led  to  mixed  results.  Their  research  concluded  that 
the  overall  sample  estimating  relationships  should  be  the  primary  equations,  with  the 
fighter  relationships  serving  as  a  complement  to  the  full  set. 

Second,  as  a  recommendation  of  the  Large,  Campbell,  and  Cates  study  to  examine 
the  effects  of  program  structure  and  airframe  construction  characteristics  as  independent 
variables  in  the  regression  model,  Hess  and  Romanoff  divided  the  independent  variables 
under  consideration  for  the  regression  models  into  four  categories:  size,  perfonnance, 
construction,  and  program.  Due  to  concerns  about  collinearity,  they  limited  each  CER  to 
only  one  variable  from  each  category  for  a  total  of  four  potential  independent  variables. 
While  many  of  the  variables  in  the  construction  and  program  categories  were  found 
significant  during  the  regression  analysis,  none  were  included  in  any  of  the  final 
equations;  Hess  and  Romanoff  detennined  that  the  addition  of  the  variables  did  not  give 
any  substantial  improvement  to  the  quality  of  the  equations. 

Finally,  an  attempt  to  add  a  technology  index  to  help  account  for  the  ever- 
increasing  complexity  of  aircraft  was  explored.  The  technology  index  was  explored  only 
within  the  fighter  category.  The  equations  of  the  technology  index  were  compared  with 
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alternate  equations  that  did  not  include  a  technology  index.  Based  on  standard  error 
comparisons,  it  was  determined  that  the  inclusion  of  a  technology  index  had  little  benefit 
for  the  fighter  airframe  CERs. 

In  summary,  Hess  and  Romanoff  updated  the  cost  estimating  relationships  for 
aircraft  airframes.  During  this  update,  no  new  explanatory  variables  were  found.  In 
addition,  attempts  to  find  equations  for  homogeneous  categories  of  aircraft  were  largely 
unsuccessful,  as  were  attempts  to  include  a  technology  index  to  account  for  increasing 
aircraft  complexities. 

Rester,  Rogers,  and  Hess:  1991 

Advanced  Airframe  Structural  Materials:  A  Primer  and  Cost  Estimating 
Methodology  expanded  the  variables  previously  considered  in  estimating  airframe  costs 
to  examine  the  effects  of  advanced  materials.  Specifically,  advanced  composite  materials 
and  new  metal  alloys  were  examined.  Rester,  Rogers,  and  Hess  (1991)  used  a  unique 
approach  to  their  study,  as  a  survey  of  industry  was  used  as  the  basis  for  their  analysis 
instead  of  the  traditional  statistical  approach. 

The  expansion  to  include  advanced  materials  as  a  consideration  in  airframe  costs 
was  driven  by  the  growing  percentage  of  these  materials  being  used  in  new  aircraft  (see 
Figure  3). 
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Figure  3.  Composite  Material  Usage  (Resetar,  Rogers,  and  Hess,  1991:3) 

In  contrast  to  previous  studies,  only  the  lowest  levels  were  examined  for  the 
effects  of  advanced  materials.  Rester,  Rogers,  and  Hess  found  that  for  both  the  recurring 
and  non-recurring  airframe  elements,  the  cost  of  aircraft  with  these  new  materials  will 
increase.  A  factor  was  then  developed  to  account  for  the  effect  of  advanced  materials  on 
airframe  cost.  To  arrive  at  total  airframe  costs,  they  combined  this  factor  with  the 
traditional  CERs  for  airframe  cost  estimation. 

Younossi,  Kennedy,  and  Graser:  2001 

Military  Airframe  Costs:  The  Effects  of  Advanced  Materials  and  Manufacturing 
Processes  is  the  most  recent  research  the  RAND  Corporation  has  completed  on  military 
airframe  costs.  It  expands  upon  the  1987  Hess  and  Romanoff  study  and  the  1991  Rester, 
Rogers,  and  Hess  studies.  “This  study  both  updates  and  extends  these  earlier  studies, 
focusing  on  the  effects  of  material  mix,  manufacturing  technique,  and  part  geometric 
complexity  on  cost”  (Younossi,  Kennedy,  and  Graser,  200 1  :xiii).  Unlike  the  Rester, 
Rogers,  and  Hess  study,  which  relied  upon  industry  surveys  as  a  primary  source  of  data, 
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this  study  used  actual  data  obtained  from  the  Multi- Aircraft  Cost  Data  and  Retrieval 
(MACDAR)  database  to  perfonn  statistical  regression  analysis.  A  final  CER 
recommended  for  estimating  fighter/attack  airframe  costs  was  derived  through  the 
compilation  of  four  regression  equations.  These  equations  factored  in  the  effects  of 
advanced  materials  and  manufacturing  processes.  In  addition,  they  allow  for  more 
flexibility  than  the  earlier  studies,  as  lot  size  and  cumulative  quantity  were  included  as 
independent  variables  in  the  model.  In  summary,  Younossi,  Kennedy,  and  Graser 
presented  a  CER  for  aircraft  airframe  that  incorporated  the  increasing  manufacturing  and 
material  complexities  of  next  generation  aircraft. 

Summary 

This  chapter  has  focused  on  gaining  an  understanding  of  the  concepts  behind  the 
basic  building  blocks  of  aircraft  cost  estimation.  These  include  an  explanation  of  the 
WBS  for  aircraft  and  the  role  of  cost  estimating  relationships.  Next,  the  two  ends  of  the 
spectrum  for  cost  estimation,  the  parametric  and  grass  roots  techniques,  were  addressed. 
Finally,  prior  research  conducted  in  this  area  was  examined.  Therefore,  this  research  will 
expand  upon  the  prior  research  by  taking  a  statistical  approach  to  the  risks  and  validity 
associated  with  a  macro  versus  micro  approach  to  aircraft  cost  estimation. 
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III.  Methodology 


Chapter  Overview 

This  chapter  provides  a  detailed  synopsis  of  the  methodology  applied  to  this 
research.  First,  an  overview  of  the  data  sources,  the  data  segregation  rationale,  and  its 
limitations  are  explored.  Second,  an  explanation  of  the  independent  and  dependent 
variables  of  the  regression  models  is  examined.  Finally,  a  summary  of  the  application  of 
statistical  processes  in  this  research,  including  multiple  regression,  Monte  Carlo 
simulation,  and  the  Bootstrap  technique  are  explained. 

Methodology  Overview 

The  analysis  begins  by  segregating  the  aircraft  cost  data  into  the  aircraft  category 
subsets  of  fighter  and  inter-theater  airlift,  and  by  their  macro  and  micro  components  of 
Flyaway  cost,  Basic  Airframe,  and  Other  Air  Vehicle.  Next,  multiple  regression 
equations  are  developed  for  each  of  these  categories,  six  in  total.  A  Monte  Carlo 
simulation  is  then  applied  to  these  regression  equations.  Specifically,  the  Bootstrap 
technique  is  used  to  estimate  the  standard  error  of  the  equations.  The  resulting 
distribution  from  the  differences  of  the  standard  error  of  the  micro  (Basic  Airframe  and 
Other  Air  Vehicle)  versus  macro  (Flyaway  Cost)  equations  is  analyzed  to  answer  the 
original  research  questions. 

Data 

Total  cost  and  component  cost  data  for  aircraft  are  required  for  this  micro  versus 
macro  analysis.  Two  primary  sources  were  used  to  gather  data.  The  main  source  of  data 
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is  the  Cost  Estimating  System  Volume  2,  Aircraft  Cost  Handbook,  Book  1 :  Aircraft,  Nov 
87,  which  was  prepared  for  the  AFCAA  by  Delta  Research  Corporation.  This  data  source 
provided  information  on  the  F-15,  F-16,  F-18,  B-l,  C-5,  C-130,  and  C-141.  The  Delta 
Research  Corporation  generated  the  data  for  their  study  through  interaction  with  the 
SPOs,  Contractor  Cost  Data  Reports  (CCDRs),  and  their  associated  contractors.  In 
addition  to  the  data  gathered  through  the  Delta  Research  Corporation  study,  data  was 
collected  directly  from  the  SPOs  for  aircraft  under  consideration  that  were  not  included  in 
the  study.  This  applies  to  the  data  from  the  C-17. 

The  primary  benefit  of  using  the  data  from  the  Delta  Research  Corporation  is  that 
it  is  normalized  to  constant  year  1987  dollars.  The  C-17  data  was  manually  adjusted 
through  the  use  of  the  Office  of  the  Secretary  of  Defense  (OSD)  inflation  indices  to 
normalize  to  constant  year  1987  dollars.  This  normalization  provides  a  homogeneous 
database  for  the  purpose  of  analysis. 

Although  both  recurring  and  non-recurring  cost  data  are  available,  only  the 
recurring  data  will  be  used  for  this  analysis.  Recurring  costs  are  costs  that  are  incurred 
on  an  ongoing  basis,  such  as  final  assembly,  while  non-recurring  costs  are  made  up  of 
one-time  expenses  such  as  initial  tooling  and  production  planning.  Because  these  two 
categories  are  influenced  by  different  sets  of  predictors,  they  are  typically  estimated 
separately  by  cost  analysts  (Seibel,  2002).  Not  separating  them  for  this  analysis  would 
add  unnecessary  variance  to  the  results,  hampering  a  comparison  of  the  macro  and  micro 
techniques  (Seibel,  2002). 

To  facilitate  the  analysis,  the  data  is  segregated  into  two  distinct  categories,  based 
on  aircraft  type,  to  achieve  homogeneity  in  the  data  sample.  The  two  categories  are 
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fighters  and  inter-theater  airlift.  The  fighter  category  is  comprised  of  the  F-15,  F-16, 
F-18,  and  B-l.  The  inter-theater  airlift  category  consists  of  the  C-17,  C-5,  C-130,  and 
C-141.  In  addition  to  segregation  by  category,  the  data  will  also  be  subdivided  by  WBS 
level.  This  WBS  breakdown,  as  described  in  Chapter  two,  will  consist  of  Flyaway  cost 
which  is  analogous  to  level  one  of  the  WBS.  The  two  analogous  components  for  WBS 
level  two  are  the  Basic  Airframe  costs  and  the  Other  Air  Vehicle  costs. 

Data  Limitations 

There  are  two  limitations  with  this  data.  The  major  limitation  is  that  the  majority 
of  the  data  is  from  pre  1987.  This  is  due  to  the  limited  availability  of  the  Delta  Research 
Corporation  Database.  Since  there  is  not  much  data  available  from  newer  systems  such 
as  the  F-22  or  Joint  Strike  Fighter,  this  is  not  a  debilitating  limitation.  However,  future 
research  would  benefit  from  obtaining  additional  data  points  from  more  recent  history. 
The  other  limitation  results  from  the  WBS  level  breakdown.  Once  again,  due  to  the 
available  data,  a  comparison  between  level  one  and  level  two  of  the  WBS  is  analyzed. 
Practitioners  may  object  that  cost  estimation  does  not  normally  occur  at  level  one.  Thus, 
future  research  may  want  to  look  at  a  different  database  that  can  be  broken  down  for  a 
level  two  versus  level  three  comparison. 

Variables 

The  development  of  high  fidelity  CERs  is  crucial  to  making  an  accurate  micro 
versus  macro  cost  comparison.  The  variables,  especially  the  independent  variables 
selected,  play  a  critical  role  in  this  CER  development  process.  The  dependent  variable  is 
cost;  as  the  goal  of  this  research  is  to  determine  whether  there  is  a  difference  in  the 
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resulting  cost  estimates  based  on  the  approach  taken.  As  detailed  in  Chapter  two, 
previous  research  has  demonstrated  that  perfonnance  parameters  are  the  most  useful  and 
accurate  independent  variables  used  for  aircraft  CERs  (AFSC  Cost  Estimating  Handbook, 
1986:6. 1).  Additionally,  the  RAND  Corporation  has  published  several  studies  that 
indicate  that  weight  and  speed  are  the  most  important  variables  for  aircraft  CERs  (Hess 
and  Romanoff,  1987:v).  Therefore,  performance  parameters,  physical  characteristics,  and 
technical  variables  will  all  be  considered  as  independent  variables  in  developing  the 
aircraft  CERs  to  ensure  a  robust  model.  The  independent  variables  that  are  investigated 
for  inclusion  in  the  model  are  found  below  (see  Table  6): 


Table  6.  Independent  Variables  Considered  for  CER  Development 


Dimensions 

(Ft) 

Weight 

(Pounds) 

Engines 

Performance 

Fuel 

Quantity 

Wing  Span 

Airframe 

Unit 

Weight 

Number 

of 

Engines 

Takeoff 

Weight 

Max  Fuel 
Internal 

Lot 

Number 

Wing  Area 
(Sq  Ft) 

Empty 

Weight 

Max 

Static 

Thrust 

Sea 

Level 

Takeoff  Run 
S.L.  (Ft) 

Max  Fuel 
External 

Cumulative 

Quantity 

Length 

Max  Gross 

Takeoff 

Weight 

Max  Speed 

S.L.  (Kts) 

Height 

Max 

Landing 

Weight 

Max  Altitude 
(Kts) 

Tread 

Wheelbase 

Regression 

A  multiple  regression  methodology  will  be  utilized  to  develop  the  aircraft  CERs. 
In  total,  six  regression  equations  will  be  developed  in  the  form  of: 
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Y=p0+piX1+p2X2  +  E  (1) 


where  Y  is  the  dependent  variable  (cost),  P  is  the  regression  coefficient,  X  is  the 
independent  variables,  and  s  is  the  error  term.  The  six  regression  equations  consist  of  a 
Flyaway  Cost,  Basic  Airframe,  and  Other  Air  Vehicle  equation  for  each  of  the  two 
categories:  fighters  and  inter-theatre  airlift. 

Independent  variables  will  be  included  in  the  model  based  on  several  diagnostic 
tests.  First,  any  multi-collinearity  problems  between  two  independent  variables  will  be 
examined.  A  multivariate  plot  of  the  independent  variables  serves  as  a  descriptive 
measure  to  detennine  if  multi-collinearity  exists  (Makridakis,  Wheelwright,  and 
Hyndman,  1998:246).  Additionally,  an  r-value  above  0.4  is  considered  a  significant 
indicator  of  multi-collinearity  and  thus  serves  as  the  diagnostic  test.  Once  multi- 
collinearity  issues  have  been  resolved  in  the  model,  the  independent  variable’s  t-statistic 
and  associated  p-values  will  be  examined.  The  p-value  associated  with  the  t-statistic 
gives  the  probability  of  attaining  the  t-statistic  under  a  true  null  hypothesis,  which  in  this 
case  is  that  the  regression  coefficient  is  zero  (Makridakis,  Wheelwright,  and  Hyndman, 
1998:217).  Thus,  a  small  p-value  indicates  there  is  little  probability  that  the  t-statistic 
could  be  its  current  value  with  a  true  null  hypothesis.  An  independent  variable  will 
therefore  be  considered  significant  if  its  p-value  is  less  than  0.10  (Makridakis, 
Wheelwright,  and  Hyndman,  1998:217).  The  Variance  Inflation  Factor  (VIF)  is  the  final 
diagnostic  that  will  be  applied  to  the  independent  variables.  A  high  VIF  score  indicates 
the  existence  of  multi-collinearity.  The  rule  of  thumb  is  to  accept  a  VIF  score  under  ten 
(www.uky.edu/ComputingCenter/SSTARS/MulticollinearityinLogisticRegression.htm). 
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The  overall  model  will  be  tested  for  predictive  capability  with  an  F-Test  and  its 
associated  p-value.  The  F-Statistic  is  the  ratio  of  the  mean  square  for  regression  and  the 
mean  square  for  error  (Makridakis,  Wheelwright,  and  Hyndman,  1998:252).  Under  a  true 
null  hypothesis,  which  states  all  regression  coefficients  are  zero,  both  the  mean  square  for 
regression  and  the  mean  square  for  error  estimate  the  variance  of  the  error  component. 
Thus,  a  large  F-statistic  indicates  that  the  model  is  explaining  the  majority  of  the  variance 
(Makridakis,  Wheelwright,  and  Hyndman,  1998:252).  The  p-value  associated  with  this 
F-statistic  will  be  deemed  significant  if  it  is  less  than  0.10.  A  second  diagnostic  measure 
used  to  detennine  whether  the  overall  model  is  acceptable  is  the  R  value.  R"  gives  the 
proportion  of  variance  in  the  dependent  variable  that  can  be  explained  by  the  independent 
variables.  In  addition  to  R  ,  the  adjusted  R~  value  is  also  examined.  Unlike  R  ,  which 
increases  with  each  additional  independent  variable  added,  adjusted  R“  penalizes  the 
regression  model  for  each  additional  independent  variable  (Devore,  2000:566).  Thus,  the 
adjusted  R  value  is  a  more  robust  indicator  of  the  models  validity. 

Once  the  independent  variables  and  the  final  model  have  been  selected,  the  three 

underlying  assumptions  of  independence,  nonnality,  and  constant  variance  are  tested. 

2 

Given  that  the  resulting  regression  model  satisfies  the  assumptions,  F-test,  t-tests,  R 
diagnostic,  and  adjusted  R  diagnostic,  it  will  be  deemed  acceptable.  The  resultant  six 
CER  equations  will  then  be  ready  for  progression  into  the  next  phase,  Monte  Carlo 
simulation. 
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Monte  Carlo  Simulation 


After  the  development  of  the  regression  equations  is  complete,  the  use  of  a  Monte 
Carlo  method  is  applied.  The  tenn  Monte  Carlo  is  very  generic,  as  it  can  be  applied  to  a 
multitude  of  differing  methods  (Woller,  1996:1).  “In  a  Monte  Carlo  method,  the  quantity 
to  be  calculated  is  interpreted  in  a  stochastic  model  and  subsequently  estimated  by 
random  sampling”  (Niederreiter,  1992:3).  Therefore,  for  an  experiment  to  be  considered 
a  Monte  Carlo  experiment,  it  simply  involves  the  use  of  random  numbers  to  examine  a 
problem.  This  technique  can  be  applied  to  a  variety  of  problems.  Figure  4  depicts  the 
general  application  of  the  Monte  Carlo  technique. 
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The  Monte  Carlo  Simulation  will  generate  multiple  outcomes  for  the  regression 
equations  for  Basic  Airframe,  Other  Air  Vehicle,  and  Flyaway  costs.  Commercially 
available  software,  Crystal  Ball,  is  implemented  to  accomplish  this  simulation.  The  use 
of  Crystal  Ball  necessitates  defining  a  Forecast  cell  and  Assumption  cells.  The  Forecast 
cell  represents  the  outcome  of  interest,  in  this  case  the  individual  regression  equation. 

The  Assumption  cell  represents  the  changing  independent  variable.  The  error  terms  from 
the  regression  equations  must  be  modeled  as  random  variables  with  a  probability 
distribution.  These  errors  will  follow  a  Normal  (0,  a“)  distribution,  and  will  constitute 
the  Assumption  cell  in  Crystal  Ball.  “To  perform  simulation  in  a  spreadsheet,  we  must 
first  place  a  random  number  generator  (RNG)  formula  in  each  cell  that  represents  a 
random,  or  uncertain,  independent  variable.  Each  RNG  provides  a  sample  observation 
from  an  appropriate  distribution  that  represents  the  range  and  frequency  of  possible 
values  for  the  variable”  (Ragsdale,  2001:567). 

Bootstrap 

The  Bootstrap  technique  and  “resampling”  are  intrinsically  tied  together.  The 
underlying  construct  behind  Bootstrap  resampling  is  that  the  original  sample  is 
considered  the  best  estimate  of  the  population  (Stine,  2002:Chapter  1 :9).  The  resampling 
occurs  as  one  “samples  the  sample”  (Stine,  2002:Chapter  1:9).  Thus,  the  essence  of  the 
Bootstrap  technique  is: 

That  in  many  complex  situations,  where  bootstrap  statistics  are  awkward  to 
compute,  they  may  be  approximated  by  Monte  Carlo  ‘resampling.’  That  is,  same- 
size  resamples  may  be  drawn  repeatedly  from  the  original  sample,  the  value  of  a 
statistic  computed  for  each  individual  resample,  and  the  bootstrap  statistic 
approximated  by  taking  an  average  of  an  appropriate  function  of  these  numbers 
(Hall,  1992:1). 
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The  following  example  illustrates  how  a  simple  bootstrap  sample  is  constructed.  It  is  of 
importance  to  note  that  the  sampling  occurs  with  replacement. 


Original  process 

Population  — >  (xi,  X2,  .. 

.,  xn) 

Resampling  process 

BS  Sample  1:  (X3,  x7, .. 

.,  X2)  ^  X  i 

BS  Sample  2:  (xg,  xi, .. 

.,  Xi)  ->  X*2 

BS  Sample  B:  (x4,  x%  .. 

Xu)  — »  X  b 

Figure  5.  Constructing  Bootstrap  Samples  (Stine,  2002:Chapter  1:10) 

The  Bootstrap  technique  is  widely  used  with  regression  equations.  Previous 
research  on  estimating  the  standard  error  of  multiple  regression  equations  found  “model- 
based  resampling  will  give  adequate  results  for  standard  error  calculations”  (Davison  and 
Hinkley,  1997:276).  The  specific  regression  resampling  approach  required  for  this 
research  is  the  Fixed  X,  residual  resampling  (Stine,  2002:Chapter  3:11).  This  approach, 
as  proposed  by  Stine,  is  a  two-step  process.  First,  a  regression  model  must  be  fit  and  the 
residuals  computed.  Second,  the  bootstrap  data  is  generated  by 

Y*  =  (Fit)  +  (BS  sample  of  OLS  residuals)  (2) 
where  Y*  is  the  dependent  variable,  and  Fit  is  the  Fixed  X  portion  of  the  regression 
equation.  It  is  important  to  note  that  under  this  method,  the  “residual  resampling  keeps 
the  same  X’s  in  every  bootstrap  sample”  (Stine,  2002:Chapter  3:23). 

Application  of  The  Bootstrap,  Monte  Carlo,  and  Crystal  Ball 

The  idea  of  a  Bootstrap  is  to  estimate  a  characteristic  (%*)  of  a  population 
distribution,  such  as  the  standard  deviation  or  mean,  “by  resampling  from  a  distribution 
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determined  by  the  original  sample  %”  (Hall,  1992:7).  Monte  Carlo  techniques  and 
Crystal  Ball  can  be  used  in  combination  to  apply  this  bootstrap  technique. 

The  statistic  of  interest  for  the  macro  versus  micro  comparison  in  this  research  is 
the  standard  error  of  the  regression  equation.  Using  the  Bootstrap  function  in  Crystal 
Ball,  the  regression  equation  as  the  Forecast  cell,  and  the  residual  Normal  (0,  a2) 
distribution  as  the  Assumption  cell,  the  standard  error  can  be  calculated  for  each 
equation.  “As  a  rule  of  thumb,  about  200  samples  are  needed  for  finding  a  standard 
error”  (Stine,  2002:  Chapter  1:16). 

Drawing  Conclusions 

The  distribution  resulting  from  the  pairing  of  the  data  points  from  the  bootstrap 
results  will  be  examined.  An  analysis  of  this  distribution,  to  include  the  mean  and  a  95% 
confidence  interval  around  the  mean,  will  be  utilized  to  determine  if  the  mean  is 
significantly  different  from  zero.  If  it  is  not  different  from  zero,  it  can  be  concluded  that 
the  error  of  the  two  equations  are  statistically  equivalent.  If  the  means  are  statistically 
different,  it  can  be  concluded  that  there  are  different  risks  from  taking  a  macro  versus 
micro  approach  to  cost  estimation.  Analysis  of  these  risks  at  the  various  WBS  levels 
enables  decisions  about  appropriate  allocation  of  resources  to  be  made.  Specifically,  it 
will  be  possible  to  detennine  whether  more  resources  should  be  allocated  to  the  Basic 
Airframe  or  to  the  Other  Air  Vehicle  category. 

Summary 

This  chapter  has  focused  on  the  underlying  methodology  necessary  to  conduct  a 
micro  versus  macro  cost  estimation  comparison.  The  origins  of  the  data,  segregation  of 
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the  data,  and  its  limitations  were  explored.  The  techniques  used  to  analyze  the  data, 
including  multiple  regression,  Monte  Carlo  Simulation,  and  the  Bootstrap  technique  were 
explained.  Chapter  four  will  take  this  theoretical  methodology  and  apply  it  to  the  real 
world  data  in  order  to  perform  the  analysis. 
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IV.  Findings  and  Discussion 


Chapter  Overview 

This  chapter  encompasses  the  analysis  of  data  as  outlined  in  the  methodology 
chapter.  Multiple  regression  models  are  developed  for  the  macro  and  micro  portions  of 
the  Fighter  and  Inter- theatre  Airlift  categories.  The  Bootstrap  technique  is  then  applied  to 
the  residuals  that  result  from  the  regression  models.  Taking  the  difference  of  the  paired 
data  gives  a  distribution  that  can  be  analyzed  to  draw  conclusions.  The  chapter  begins 
with  an  analysis  of  the  Fighter  category  data,  followed  by  an  analysis  of  the  Inter- theatre 
Airlift  category  data. 

Multiple  Regression  Models  for  the  Fighter  Category 

Data 

The  data  for  the  Fighter  category  comes  from  the  Delta  Research  Corporations 
report,  as  detailed  in  the  previous  chapter.  The  four  aircraft  under  consideration  are  the 
F-15,  F-16,  F-18,  and  B-l.  These  aircraft  were  chosen  for  three  reasons.  First,  they  are 
all  operational  aircraft  that  are  currently  utilized  by  their  respective  service.  Second, 
multiple  production  data  points  are  available  for  analysis.  Multpile  data  points  enhance 
the  probability  of  generating  a  robust  model,  which  is  imperative  for  conducting  the 
regression  analysis.  It  is  important  to  note  that  this  condition  eliminated  next  generation 
aircraft  such  as  the  F-22  or  Joint  Strike  Fighter,  which  do  not  have  production  data. 

Third,  the  characteristics  of  these  aircraft  provide  a  natural  grouping  that  allows  for  a 
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homogeneous  database  to  analyze.  The  final  database  consisting  of  the  four  aircraft  has 
47  data  points. 

Preliminary  Modeling  Problem 

Initial  development  of  the  regression  models  included  consideration  of  all  the 
independent  variables  listed  in  Table  6  (see  Page  37).  As  shown  below  (see  Table  7) 
with  a  portion  of  the  F-16  data,  there  is  duplicity  in  many  of  the  independent  variables. 
For  example,  although  the  average  lot  cost  decreases  as  subsequent  lot  buys  occur,  the 
wing  area  remains  constant  at  300  square  feet.  While  the  learning  curve  effect  is 
captured  with  variables  such  as  cumulative  quantity  and  lot  size,  a  bias  is  introduced  into 
the  regression  by  the  duplicate  independent  variables. 


Table  7.  Portion  of  F-16  Independent  Variables  Data 


A/C 

Lot 

# 

Lot 

Qty. 

Avg. 

Lot 

Cost 

Wing 

Span 

Wing 

Area 

(Sq 

Ft) 

Length 

Height 

Tread 

Wheel 

-base 

F-16 

1 

8 

23.66 

32.8 

300 

49.4 

16.4 

7.8 

13.1 

F-16 

2 

55 

7.84 

32.8 

300 

49.4 

16.4 

7.8 

13.1 

F-16 

3 

105 

9.02 

32.8 

300 

49.4 

16.4 

7.8 

13.1 

F-16 

4 

145 

713 

32.8 

300 

49.4 

16.4 

7.8 

13.1 

F-16 

5 

75 

6.74 

32.8 

300 

49.4 

16.4 

7.8 

13.1 

F-16 

6 

348 

7.44 

32.8 

300 

49.4 

16.4 

7.8 

13.1 

F-16 

7 

175 

5.28 

32.8 

300 

49.4 

16.4 

7.8 

13.1 

F-16 

8 

180 

5.23 

32.8 

300 

49.4 

16.4 

7.8 

13.1 

F-16 

9 

160 

5.11 

32.8 

300 

49.4 

16.4 

7.8 

13.1 

F-16 

10 

120 

6.68 

32.8 

300 

49.4 

16.4 

7.8 

13.1 

F-16 

11 

144 

5.85 

32.8 

300 

49.4 

16.4 

7.8 

13.1 

F-16 

12 

150 

6.26 

32.8 

300 

49.4 

16.4 

7.8 

13.1 

There  are  two  potential  solutions  to  this  problem.  First,  changes  in  the 
performance  parameters  and  physical  characteristics  occur  as  the  aircraft  changes  (i.e. 
from  an  F-15  to  an  F-16)  and  as  the  aircraft  model  changes.  For  instance,  when  the  F-15 
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was  updated  to  the  C  model,  the  maximum  internal  fuel  characteristic  changed.  Thus, 
one  way  to  model  the  regression  is  to  make  a  qualitative  independent  variable  that 
represents  an  aircraft  that  has  the  same  performance  parameters  and  physical 
characteristics.  The  learning  curve  portion  of  the  regression  model  would  still  be 
captured  through  independent  quantity  variables.  A  major  benefit  to  this  approach  is  that 
all  47  data  points  would  remain  in  the  model.  The  major  detriment  to  this  approach  is 
that  the  independent  variables  may  not  be  meaningful  to  the  practitioner.  However,  it  is 
important  to  note  that  the  objective  is  not  to  have  a  practitioner  use  the  regression 
equations,  but  rather  to  achieve  the  best  estimate  of  the  standard  error  of  the  regression 
equation  for  comparative  purposes. 

The  second  option  would  be  to  only  use  one  data  point  from  each  aircraft  at  a 
specific  quantity,  such  as  100.  This  option  would  alleviate  the  bias  found  in  the 
independent  variables.  However,  this  approach  would  result  in  a  regression  model  with 
only  four  data  points.  Therefore,  the  number  of  independent  variables  would  be  limited 
to  two,  due  to  the  degrees  of  freedom  in  the  regression  model.  The  primary  benefit  of 
this  approach  is  that  the  regression  equation  would  be  useful  to  a  practitioner.  However, 
there  are  some  significant  problems  with  this  approach.  Preliminary  models  using  this 
technique  found  that  the  B-l  was  a  highly  influential  data  point.  Leaving  this  data  point 
in  the  model  may  invalidate  the  results  of  the  regression,  including  the  p-values 
associated  with  the  independent  variables,  the  assumptions,  and  the  regression 
coefficients  (Makridakis,  Wheelwright,  and  McGee,  1983:  213). 

To  achieve  the  objectives  of  a  comparison  of  the  micro  and  macro  approaches  to 
cost  estimation,  the  validity  of  the  errors  resulting  from  the  regression  models  must  be  of 
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the  highest  quality.  Therefore,  the  first  solution  of  using  qualitative  variables  is  the 
preferred  solution  to  this  problem.  This  method  provides  a  mathematical  model  that  best 
estimates  the  errors.  For  comparative  purposes  only,  the  second  method’s  model  for  the 
Basic  Airframe  is  developed  and  attached  as  Appendix  A. 

Basic  Airframe  Model 

The  variable  of  interest  in  this  model  is  the  cost  of  fighter  aircraft.  The  first  step 
in  any  regression  model  is  to  ensure  the  response  is  continuous.  A  histogram  of  the 
dependent  variable  can  be  utilized  for  this  purpose  (see  Figure  6).  Although  there  are 
small  gaps  in  the  data,  it  still  appears  to  be  relatively  continuous.  In  addition,  the  data 
does  not  appear  to  be  gathering  in  bins.  Thus,  these  characteristics  lead  to  the  conclusion 
that  the  data  is  continuous. 


$M 

i 

1  2  3  4  5  6 


Figure  6.  Continuous  Data  Histogram  for  Basic  Airframe 

Once  the  data  is  deemed  continuous,  preliminary  regression  analysis  is  conducted. 
The  independent  variables  used  are  Category  (due  to  the  reasons  explained  above)  and 
quantity  variables  including  Lot  #,  Lot  Size,  and  Cumulative  Quantity.  Running  the 
initial  regression  model  revealed  significant  independent  variables  but  violated  the 
assumptions  of  constant  variance  and  normality.  Looking  at  the  residual  by  predicted 
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plot,  the  constant  variance  violation  is  apparent  (see  Figure  7).  There  is  a  clear 
megaphone  pattern  in  this  plot.  Such  a  trend  is  an  indication  that  constant  variance  is 
violated  (Neter,  Kutner,  Nachtsheim,  and  Wasserman,  1996:102).  The  normality 
violation  is  shown  with  a  plot  of  the  studentized  residuals.  The  bell-shaped  line  is  the 
fitted  nonnal  distribution.  In  addition,  the  Shapiro-Wilk  p-value  associated  with  the 
residuals  is  <.0001  leading  to  a  rejection  of  the  null  hypothesis  (Neter,  Kutner, 
Nachtsheim,  and  Wasserman,  1996: 1 1 1).  Clearly,  the  data  is  not  normally  distributed. 


Figure  7.  Assumption  Violation  Plots  for  Basic  Airframe 


The  solution  to  these  assumption  violations  is  to  take  the  natural  log  of  the 
dependent  (Y)  variable  (Neter,  Kutner,  Nachtsheim,  and  Wasserman,  1996:129).  This 
approach  is  consistent  with  the  methodology  applied  by  the  RAND  studies  discussed  in 
Chapter  2.  For  consistency  in  comparison,  all  the  regression  models  in  the  Fighter 
category  will  use  the  natural  log  of  Y  as  the  dependent  variable. 

The  full  model  is  now  run  again  with  ln(Y)  as  the  response.  All  data  points  are 
included  in  the  initial  model.  Through  several  iterations  of  searching  for  the  best  model 
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based  on  p-values,  VIF  scores,  outliers,  influential  data  points,  and  R"  values,  a  final 
model  is  derived.  The  Analysis  of  Variance  (ANOVA)  table  is  shown  in  Figure  8.  The 
F-Ratio  is  50.95  and  its  associated  p-value  is  <.0001.  The  p-value  gives  the  probability 
of  attaining  an  F  statistic  as  large  as  50.95,  given  the  slope  of  the  regression  coefficients 
is  zero.  The  p-value  is  less  than  the  a  =  0. 10  threshold,  so  it  can  be  concluded  that  at 
least  one  of  the  parameters  in  the  model  has  predictive  capability. 


Table  8.  ANOVA  Table  for  Basic  Airframe 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Model 

7 

48.297758 

6.89968 

50.9514 

Error 

38 

5.145845 

0.13542 

Prob  >  F 

C.  Total 

45 

53.443603 

<.0001 

Influential  data  points  can  skew  regression  model  output.  Cook’s  Distance 
Influence  Test  (Cook’s  D)  can  be  used  to  detennine  if  influential  data  points  are  present 
in  a  regression  model  (Neter,  Kutner,  Nachtsheim,  and  Wasserman,  1996:380).  This  test 
operates  by  removing  each  data  point  from  the  data  set,  one  at  a  time,  and  producing  a 
number  associated  with  that  points’  influence.  If  the  resultant  number  is  between  0  and 
0.25  then  the  point  has  little  apparent  influence  on  the  fitted  values.  If  the  number  is 
between  0.25  and  0.50  it  is  considered  a  mild  influential  data  point.  However,  if  the 
number  is  greater  than  0.50  then  that  point  has  a  major  influence  on  the  fit  (Neter, 

Kutner,  Nachtsheim,  and  Wasserman,  1996:381).  A  visual  plot  of  Cook’s  D  values  (see 
Figure  8)  indicates  that  there  are  no  influential  data  points  in  this  regression  model,  as  the 
highest  Cook’s  D  value  is  0.229. 
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Figure  8.  Cook's  D  Influence  for  Basic  Airframe 


Since  there  are  no  influential  data  points  in  the  regression  model,  it  is  appropriate 
to  look  at  the  individual  parameter  estimate  values  to  ensure  they  are  significant  (Neter, 
Kutner,  Nachtsheim,  and  Wasserman,  1996:232).  An  a  =  0.10  threshold  level  is  utilized. 
Analyzing  the  independent  variables  in  Table  9,  all  p-values  are  significantly  below  the 
threshold  level.  Therefore,  it  can  be  concluded  that  all  parameters  in  the  model  are 
significant. 


Table  9.  Parameter  Estimates  for  Basic  Airframe 


Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

Intercept 

3.3941178 

0.136297 

24.90 

<.0001 

Category[l] 

-0.709052 

0.153275 

-4.63 

<.0001 

Category[21 

-0.534294 

0.139719 

-3.82 

0.0005 

Category[3] 

-1.451749 

0.130112 

-11.16 

<.0001 

Category[41 

-0.891017 

0.321027 

-2.78 

0.0085 

Category[5] 

-0.25603 

0.137962 

-1.86 

0.0713 

Category[6] 

2.0370961 

0.339241 

6.00 

<.0001 

Cum  Qty 

-0.000796 

0.000243 

-3.28 

0.0022 

Another  important  diagnostic  to  determine  the  appropriateness  of  the  goodness  of 

2  *22* 

fit  of  the  model  is  the  R“  value  and  the  adjusted  R  value.  R“  gives  the  portion  of  Y  that 
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is  explained  by  the  X’s.  One  drawback  to  R~  is  that  its  value  can  be  artificially  inflated 
by  adding  additional  X  variables.  To  counter  this  effect,  adjusted  R  is  often  used  in  its 
place.  Adjusted  R"  penalizes  the  model  for  each  additional  X  variable  added  (Neter, 
Kutner,  Nachtsheim,  and  Wassennan,  1996:230).  Table  10  gives  the  Summary  of  Fit 
results  for  the  Basic  Airframe  model.  The  R2  value  of  0.90  is  extremely  high,  as  is  the 
adjusted  R2  value  of  0.886.  These  results  mean  that  90%  of  the  variability  in  the  model 
can  be  explained  by  the  independent  variables.  This  is  an  extremely  high  result  and 
supports  the  conclusion  that  this  is  a  good  model. 


Table  10.  Summary  of  Fit  for  Basic  Airframe 


RSquare 

0.903714 

RSquare  Adj 

0.885978 

Root  Mean  Square  Error 

0.36799 

Mean  of  Response 

2.517431 

Observations  (or  Sum  Wgts) 

46 

Before  concluding  that  the  model  is  valid,  the  three  underlying  assumptions  of 
nonnality,  independence,  and  constant  variance  must  be  satisfied.  Two  diagnostics  are 
utilized  to  check  for  normality.  First,  a  histogram  of  the  studentized  residuals  is  plotted 
to  analyze  the  normality  assumption  with  a  normal  curve  imposed  over  the  histogram 
(Neter,  Kutner,  Nachtsheim,  and  Wassennan,  1996:  111).  Although  there  is  a  spike  in  the 
data,  it  appears  that  the  data  is  normally  distributed  (see  Figure  9).  Further  analysis  of  the 
spike  indicates  that  the  data  points  gathered  there  do  not  have  any  characteristics  that 
suggest  there  is  a  split  distribution  of  the  data.  This  conclusion  comes  from  an  analysis 
of  the  raw  data.  The  data  points  in  the  spike  come  from  across  all  the  aircraft  categories. 
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Figure  9.  Normality  Histogram  for  Basic  Airframe 


The  Shapiro-Wilk  test  can  be  used  as  a  quantitative  diagnostic  to  evaluate  the 
Goodness  of  Fit  of  the  Normal  Distribution.  The  null  hypothesis  of  this  test  is  that  the 
data  is  normally  distributed.  The  Shapiro-Wilk  test  produced  a  p-value  of  0.0345,  which 
is  less  than  the  a-level  threshold  of  0.05.  Although  this  value  is  less  than  the  threshold,  it 
is  not  a  significant  violation.  Therefore,  this  violation  is  deemed  a  reasonable  violation 
and  the  normality  assumption  is  satisfied. 

The  most  important  assumption  that  must  be  satisfied  is  the  assumption  of 
independence.  Typically,  the  Durbin-Watson  test  can  be  used  to  determine  if  the  model 
exhibits  independence.  The  Durbin-Watson  test  examines  the  Lag  1  correlation  of  the 
residuals  (Neter,  Kutner,  Nachtsheim,  and  Wasserman,  1996:504).  However,  the  aircraft 
data  in  this  model  exhibits  time  series  characteristics,  as  the  data  points  come  from 
production  lot  buys  over  time.  Therefore,  it  is  more  appropriate  to  look  at  the  full 
Autocorrelation  Function  rather  than  strictly  at  the  Lag  1  autocorrelation  of  the  Durbin 
Watson  Test.  Figure  10  shows  the  Autocorrelation  Function  for  the  Basic  Airframe. 
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Figure  10.  Autocorrelation  Function  for  Basic  Airframe 


The  two  solid  vertical  lines  in  Figure  10  designate  the  critical  value  lines.  These 
values  are  calculated  as  2/  -Jn  .  To  exhibit  independence,  several  characteristics  are 
desired.  First,  the  function  should  dampen  quickly.  Second,  there  should  not  be  any 
remaining  pattern  in  the  residuals.  Third,  the  majority  of  the  spikes  should  be  within  the 
critical  value  line  limits  (Makridakis,  Wheelwright,  and  McGee,  1983:388).  Clearly,  this 
criterion  is  met  with  this  model  and  thus  the  assumption  of  independence  is  satisfied. 

The  third  assumption  that  must  be  considered  is  constant  variance.  Constant 
variance  can  be  detennined  by  two  methods.  A  visual  plot  of  the  residual  by  predicted 
gives  a  descriptive  measure  for  constant  variance  (see  Figure  1 1).  Constant  variance  is 
characterized  by  the  absence  of  a  pattern  (Neter,  Kutner,  Nachtsheim,  and  Wasserman, 
1996:102). 
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Figure  11.  Residual  by  Predicted  Plot  for  Basic  Airframe 


The  Breusch-Pagan  test  is  a  quantitative  test  for  constant  variance.  This  test  is 
used  to  calculate  a  test  statistic  that  is  approximately  Chi-Square  distributed,  for  large  n, 
with  degrees  of  freedom  equal  to  the  degrees  of  freedom  of  the  model.  The  test  statistic 
is  defined  as  (Neter,  Kutner,  Nachtsheim,  and  Wassennan,  1996:115): 


X 


SSR 
df  +  1 


SSE 
V  n  J 


(3) 


The  test  statistic  for  the  Basic  Airframe  model  is  3.905,  calculated  as  follows: 


.3908 

- - -  =  3.905 

f  5,145  ']2 

{  46  J _ 

The  p-value  associated  with  this  '£  value  and  seven  degrees  of  freedom  is  0.79.  Thus  the 
null  hypothesis  is  not  rejected  and  the  assumption  of  constant  variance  is  satisfied. 
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Checking  for  outliers  is  the  final  step  before  declaring  the  model  valid.  An  outlier 
is  found  by  looking  at  the  minimum  and  maximum  quantiles  when  the  studentized 
residuals  are  fit  with  a  nonnal  distribution.  Values  outside  the  minimum  bound  of  less 
than  -3  or  the  maximum  bound  of  greater  than  3  indicates  the  presence  of  an  outlier 
(Neter,  Kutner,  Nachtsheim,  and  Wasserman,  1996:103).  The  maximum  value  for  this 
model  is  2.47  and  the  minimum  value  is  -2.29.  Therefore,  it  can  be  concluded  that  there 
are  no  outliers  in  the  model. 

The  Basic  Airframe  regression  model  is  complete  and  validated  by  the  various 
diagnostics  discussed  above.  Thus,  the  resulting  equation  from  this  model  is: 

Y=  3.394  -  0.709*CW(1)  -  0.534*CW(2)  -  1.452*CW(3)  -  0.891*CW(4)  -  0.256*CW(5) 
+2.037*CW(6)  -  0.001*  CumQty  +  £  (4) 

In  order  to  perform  the  macro  versus  micro  comparison  we  need  the  residual  term  for  the 
model.  The  underlying  distribution  of  the  residuals  is  shown  in  Figure  12.  The  residuals 
have  a  Normal  Distribution  with  p  =  0  and  a  =  0.334.  The  Kolmogorov-Smirnov  Test 
returns  a  p-value  of  >0.15,  validating  the  nonnal  distribution  fit  (Neter,  Kutner, 
Nachtsheim,  and  Wasserman,  1996: 1 1 1).  This  distribution  is  critical  as  an  input  to  the 
Bootstrap  technique  that  will  be  used  to  perform  the  micro  versus  macro  comparison. 
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Figure  12.  Plot  of  Residuals  for  Basic  Airframe 

Other  Air  Vehicle  Model 

The  second  WBS  level  two  regression  model  developed  was  the  Other  Air 
Vehicle  model.  To  begin  the  analysis,  the  dependent  variable  is  examined  to  ensure  it  is 
continuous.  As  shown  in  Figure  13,  there  does  not  appear  to  be  any  major  gaps  nor  does 
the  data  appear  to  be  gathering  in  large  bins.  Therefore,  it  is  appropriate  to  conclude  that 
the  data  is  continuous. 
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Figure  13.  Continuous  Data  Histogram  for  Other  Air  Vehicle 

As  explained  during  the  development  of  the  Basic  Airframe  model,  the 
independent  variable  used  for  analysis  will  be  the  natural  log  of  Y.  This  is  a  direct  result 
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of  the  violations  of  constant  variance  and  normality  in  the  model.  The  next  step  is  to 
examine  the  ANOVA  table  to  ensure  the  Other  Air  Vehicle  model  has  predictive 
capability.  As  shown  in  Table  11,  the  p-value  is  <.0001  indicating  that  at  least  one  of  the 
model  parameters  has  predictive  capability.  One  other  item  of  interest  from  the  ANOVA 
table  is  the  degrees  of  freedom.  The  degrees  of  freedom  for  the  model  are  only  30, 
despite  having  47  initial  data  points.  This  is  due  to  the  exclusion  of  an  independent 
variable  (Category  2)  and  some  influential  data  points.  The  reasons  behind  these 
exclusions  will  be  expanded  upon  further. 


Table  11.  ANOVA  Table  for  Other  Air  Vehicle 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Model 

5 

7.556954 

1.51139 

9.6408 

Error 

25 

3.919273 

0.15677 

Prob  >  F 

C.  Total 

30 

11.476227 

<.0001 

Cook’s  D  test  is  run  to  check  for  influential  data  points  in  the  model.  As  shown  in 
Figure  14  below,  all  the  data  points  fall  below  the  0.25  thresholds.  However,  looking  at 
the  graph,  it  appears  that  the  majority  of  the  data  points  are  gathered  in  a  very  tight  range 
with  a  couple  lingering  at  the  top  right.  Therefore,  additional  analysis  on  these  data 
points  and  the  effect  they  may  have  on  the  model  is  warranted. 
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Figure  14.  Cook's  D  Influence  for  Other  Air  Vehicle 


2 

A  comparison  of  the  R  values  and  the  p-values  associated  with  the  parameter 
estimates  of  the  independent  variables  found  that  three  of  the  data  points  were  indeed 
influencing  the  model.  This  was  detennined  by  re-running  the  model  without  the 
suspected  influential  data  points.  Removal  of  these  data  points  resulted  in  a  0.22 
improvement  in  the  adjusted  R  value.  In  addition,  all  of  the  independent  variables,  with 
the  lone  exception  of  Category  2,  are  now  significant.  The  Summary  of  Fit  and 
Parameter  Estimates  found  in  Table  12  below  shows  the  results  with  the  removal  of  the 
three  influential  data  points  and  the  Category  2  data. 
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Table  12.  Summary  of  Fit  and  Parameter  Estimates  for  Other  Air  Vehicle 


Rsquare 

0.658488 

RSquare  Adj 

0.590185 

Root  Mean  Square  Error 

0.395943 

Mean  of  Response 

1.443977 

Observations  (or  Sum  Wgts) 

31 

Summary  of  Fit 


Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

VIF 

Intercept 

2.1343389 

0.161211 

13.24 

<.0001 

Category  [11 

-0.315531 

0.164369 

-1.92 

0.0664 

1.4120392 

Category[3] 

-0.395569 

0.134653 

-2.94 

0.0070 

1.0894043 

Category[41 

0.5669808 

0.345178 

1.64 

0.1130 

3.7755367 

Category[5] 

-0.376998 

0.157462 

-2.39 

0.0245 

1.5713612 

Cum  Qty 

-0.001152 

0.000277 

-4.16 

0.0003 

3.4467635 

Parameter  Estimates 


The  independent  variables  are  deemed  significant  if  the  associated  p-value  is  less 
than  the  a  =  0. 10  level.  Clearly,  Category  4  does  not  meet  this  criterion.  However, 
because  the  p-value  of  0.1 13  is  close  to  the  threshold  and  it  is  desirable  to  have  as  many 
data  points  as  possible  in  the  model,  the  variable  is  kept  in  the  model.  Also  of  note  in 
Table  12  are  the  VIF  scores.  The  highest  VIF  score  is  3.78,  which  is  well  below  the 
target  goal  of  less  than  ten.  The  potential  problem  of  multi-collinearity  is  thus  averted  in 
this  model. 

Examining  a  plot  of  Cook’s  D  test  results  with  the  remaining  30  data  points  yields 
the  results  found  in  Figure  15  below.  The  spread  of  the  data  points  is  now  more  uniform. 
In  addition,  all  the  values  are  below  0. 15.  Clearly,  from  an  influential  data  point  view, 
this  is  now  a  better  model. 
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Figure  15.  Revised  Cook's  D  Influence  for  Other  Air  Vehicle 

Before  deeming  the  model  valid,  the  three  assumptions  of  normality, 
independence,  and  constant  variance  must  be  satisfied.  The  plot  of  the  studentized 
residuals  (see  Figure  16)  appears  to  have  the  expected  bell  shape  associated  with  a 
normal  curve.  The  Shapiro-Wilk  test  confirms  this  suspicion,  returning  a  p-value  of  0. 16. 
This  is  well  above  the  a  =  0.05  threshold.  Thus,  the  assumption  of  normality  is  satisfied. 


Figure  16.  Normality  Histogram  for  Other  Air  Vehicle 

One  final  check  that  is  accomplished  with  a  histogram  of  the  studentized  residuals 
is  to  determine  if  outliers  are  present.  As  shown  in  Figure  16,  it  appears  that  all  the  data 
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points  are  within  three  standard  deviations.  The  quartiles  table  confirms  this  visual 
assumption  as  the  maximum  and  minimum  values  fall  within  negative  three  and  three 
(see  Table  13).  Clearly,  there  are  no  outliers  in  this  model. 


Table  13.  Quartiles  for  Other  Air  Vehicle 


100.0% 

maximum 

2.521 

99.5% 

2.521 

97.5% 

2.521 

90.0% 

1.200 

75.0% 

quartile 

0.480 

50.0% 

median 

-0.049 

25.0% 

quartile 

-0.486 

10.0% 

-0.889 

2.5% 

-2.822 

0.5% 

-2.822 

0.0% 

minimum 

-2.822 

The  assumption  of  independence  is  confirmed  by  examining  the  Autocorrelation 
Plot  (see  Figure  17).  Every  lag  in  the  autocorrelation  plot  is  below  the  threshold  level. 
This  lack  of  pattern  in  the  residuals  is  a  clear  indicator  that  independence  is  not  a 
problem. 
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Figure  17.  Autocorrelation  Function  for  Other  Air  Vehicle 


The  final  assumption  is  constant  variance.  Figure  18  gives  a  plot  of  the  residuals 
versus  predicted.  While  there  is  not  an  overwhelming  pattern,  there  does  appear  to  be  a 
potential  megaphone  effect.  To  test  this  visual  observation,  the  Breusch-Pagan  test  is 
applied. 


.515 

6 

1919^| 2 
31  ) 


5.371 


The  p-value  returned  from  this  test  is  0.37,  refuting  the  visual  observation.  Therefore, 
the  assumption  of  constant  variance  is  satisfied. 
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Figure  18.  Residual  by  Predicted  Plot  for  Other  Air  Vehicle 

The  Other  Air  Vehicle  regression  model  is  complete  and  validated  by  the  various 
diagnostics  discussed  above.  Thus,  the  resulting  equation  from  this  model  is: 

Y=  2.134  -  0.316*C«t(l)  -  0.396*CW(3)  +  0.567*CW(4)  -  0.377*CW(5)  - 

0.01 1  *CumQty  +  £  (5) 

The  residual  term  is  the  item  of  interest  to  perfonn  the  macro  versus  micro 
comparison.  The  underlying  distribution  of  the  residuals  is  shown  in  Figure  19.  The 
residuals  have  a  Normal  Distribution  with  p  =  0  and  a  =  0.356.  The  Kolmogorov- 
Smimov  Test  returns  a  p-value  of  >0. 15,  validating  the  normal  distribution  fit.  This 
distribution  is  critical  as  an  input  to  the  Bootstrap  technique  that  will  be  used  to  perfonn 
the  macro  versus  micro  comparison. 
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Figure  19.  Plot  of  Residuals  for  Other  Air  Vehicle 


Flyaway  Cost  Model 

The  Flyaway  Cost  model  represents  the  macro  portion  of  the  analysis.  The 
dependent  variable  is  first  analyzed  to  ensure  it  is  continuous.  While  there  are  some 
small  gaps  in  the  data,  it  appears  to  be  continuous  (see  Figure  20). 


Figure  20.  Continuous  Data  Histogram  for  Flyaway  Cost 

Determining  whether  the  overall  model  has  predictive  capability  is  accomplished 
by  examining  the  ANOVA  table  (see  table  14).  The  p-value  is  <.0001,  indicating  that  at 
least  one  of  the  parameters  has  predictive  capability. 
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Table  14.  ANOVA  Table  for  Flyaway  Cost 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Model 

7 

37.202530 

5.31465 

44.7988 

Error 

38 

4.508085 

0.11863 

Prob  >  F 

C.  Total 

45 

41.710615 

<.0001 

A  plot  of  Cook’s  D  is  used  as  a  diagnostic  for  influential  data  points  (see  Figure 
21).  All  the  data  points  are  below  the  0.25  threshold.  In  addition,  the  spread  of  the  data 
points  is  relatively  uniform.  An  examination  of  the  effects  of  removing  the  top  right  data 
point  did  not  dramatically  affect  the  model.  Therefore,  it  is  concluded  that  there  are  no 
influential  data  points  in  the  model. 


2  •  2 

The  Summary  of  Fit  table  (see  Table  15)  returns  an  R  of  0.89  and  an  adjusted  R" 

of  0.87,  indicating  that  this  model  does  an  excellent  job  of  explaining  the  variability. 


63 


Additionally,  the  Parameter  Estimates  are  all  significantly  below  the  a  =  0. 10  threshold. 
Multi-collinearity  in  the  independent  variables  is  ruled  out  by  the  low  VIF  scores.  Thus, 
all  independent  variables  are  included  in  the  final  model. 


Table  15,  Summary  of  Fit  and  Parameter  Estimates  for  Flyaway  Cost 


RSquare 

0.89192 

RSquare  Adj 

0.87201 

Root  Mean  Square  Error 

0.344433 

Mean  of  Response 

2.859426 

Observations  (or  Sum  Wgts) 

46 

Summary  of  Fit 


Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

VIF 

Intercept 

3.7395676 

0.127571 

29.31 

<.0001 

Category  f  11 

-0.697189 

0.143462 

-4.86 

<.0001 

1.6745295 

Category[2] 

-0.431093 

0.130774 

-3.30 

0.0021 

1.9085175 

Category[31 

-1.221545 

0.121782 

-10.03 

<.0001 

1.402332 

Category[4] 

-0.548961 

0.300476 

-1.83 

0.0756 

4.5662778 

Category[5] 

-0.36779 

0.12913 

-2.85 

0.0071 

1.7722244 

Category[61 

1.7433997 

0.317524 

5.49 

<.0001 

3.3255161 

Cum  Qty 

-0.000895 

0.000227 

-3.94 

0.0003 

3.8783792 

Parameter  Estimates 


The  three  assumptions  of  normality,  independence,  and  constant  variance  must  be 
satisfied  before  accepting  the  model  as  complete.  Figure  22  shows  a  histogram  of  the 
studentized  residuals.  The  bell  shape  of  the  data  gives  a  visual  check  for  normality.  The 
more  robust  quantitative  measure,  the  Shapiro-Wilk  test,  returns  a  p-value  of  0.07.  Thus, 
the  assumption  of  normality  is  validated. 
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Figure  22.  Normality  Histogram  for  Flyaway  Cost 

The  studentized  residuals  can  also  be  used  to  check  for  outliers.  Visually,  Figure 
22  does  not  show  any  data  points  beyond  three  standard  deviations.  Table  16  verifies  this 
visual  observation,  as  the  farthest  point  is  at  -2.71.  Therefore,  it  is  concluded  that  this 
model  does  not  contain  any  outliers. 


Table  16.  Quartiles  for  Flyaway  Cost 


100.0% 

maximum 

2.175 

99.5% 

2.175 

97.5% 

2.175 

90.0% 

1.905 

75.0% 

quartile 

0.499 

50.0% 

median 

-0.198 

25.0% 

quartile 

-0.686 

10.0% 

-1.113 

2.5% 

-2.469 

0.5% 

-2.713 

0.0% 

minimum 

-2.713 

The  Autocorrelation  Plot  confirms  the  assumption  of  independence  (see  Figure 
23).  There  is  one  spike  that  exceeds  the  critical  bounds  at  Lag  2.  It  is  expected  that  five 
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percent  of  the  values  may  be  outside  the  critical  bounds  due  to  noise  in  the  data. 
Therefore,  this  spike  is  not  a  concern  and  the  assumption  of  independence  is  satisfied. 


Lag  AutoCorr  -.8 -.6 -.4 -.2  0  .2  .4  .6  .8 

0  1.0000 

1  0.2554 

2  0.5467 

3  0.1586 

4  0.1148 

5  0.0027 

6  -0.0019 

7  0.0006 

8  0.0035 

9  0.0240 

10  0.0424 

11  0.0752 

12  0.0506 

13  0.0880 

14  0.0103 


tr 


Figure  23.  Autocorrelation  Function  for  Flyaway  Cost 


The  last  assumption  to  explore  is  constant  variance.  Figure  24  shows  a  plot  of  the 

residuals  against  predicted.  Visually,  there  does  not  appear  to  be  an  overriding  pattern, 

however  there  does  appear  to  be  a  slight  megaphone  effect.  Using  the  Breusch-Pagan  as 

the  quantitative  diagnostic,  the  following  value  is  calculated: 

.2116 

8 

-  =  2.754 

f  4.508 

l  46  ) _ 

2 

The  p-value  associated  with  the  %  statistic  of  2.754  is  0.907.  Constant  variance  is 
thereby  satisfied. 
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In(Y)  $  Predicted 


Figure  24.  Residual  by  Predicted  Plot  for  Flyaway  Cost 


The  Flyaway  Cost  regression  model  is  validated  by  the  diagnostics  above.  The 
resulting  equation  is: 

Y=  3.740  -  0.697*Otf(l)  -  0.431*Otf(2)  -  1.22*Cat(3)  -  0.549*Cat(4)  -  0.368*C«t(5) 
+  1.743*Cat(6)  -  0Ml*CumQty  +  s  (6) 

The  error  term  is  the  item  of  interest  to  perform  the  macro  versus  micro 
comparison.  The  underlying  distribution  of  the  residuals  is  shown  in  Figure  25.  The 
residuals  have  a  Normal  Distribution  with  p  =  0  and  a  =  0.3 13.  The  Kolmogorov- 
Smimov  Test  returns  a  p-value  of  >0. 15,  validating  the  normal  distribution  fit.  This 
distribution  is  critical  as  an  input  to  the  Bootstrap  technique  that  will  be  used  to  perfonn 
the  macro  versus  micro  comparison. 
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Figure  25.  Plot  of  Residuals  for  Flyaway  Cost 


Bootstrap  Comparison  for  Flyaway  Versus  Basic  Airframe 

Generation  of  the  regression  equations  leads  to  the  next  step  in  the  analysis: 
Fixed  X,  residual  resampling.  Starting  with  the  Flyaway  regression  equation,  Crystal 
Ball  performs  the  bootstrap  technique.  This  is  accomplished  by  defining  the  assumption 
cell  as  a  Normal  (0,  0.3 13)  distribution.  This  distribution  represents  the  residuals  as 
determined  from  the  regression  equation.  200  bootstrap  samples  are  generated,  with 
1000  iterations  occurring  within  each  sample.  The  frequency  histogram  for  the  standard 
deviation  from  these  trials  is  shown  in  Figure  26.  As  anticipated,  the  majority  of  the 
trials  are  found  near  the  0.32  value,  with  a  relatively  tight  range  from  0.30  to  0.33. 
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Figure  26.  Frequency  Chart  of  Flyaway  Bootstrap  Data 


Next,  the  bootstrap  technique  is  replicated  for  the  Basic  Airframe  category.  The 
only  change  is  that  the  residuals  are  modeled  as  a  Normal  (0,  0.334)  distribution,  as 
determined  from  the  regression  equation.  The  resulting  bootstrap  standard  deviation 
frequency  chart  is  shown  in  Figure  27.  One  item  of  interest  from  this  chart  is  that  the 
spread  of  data  is  wider  than  the  Flyaway  Cost  range  by  0.01.  This  may  give  a 
preliminary  indication  that  there  is  a  difference  in  the  two  error  components. 
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Figure  27.  Frequency  Chart  of  Basic  Airframe  Bootstrap  Data 


The  standard  error  of  the  resulting  200  bootstrap  samples  from  the  Flyaway  and 
Basic  Airframe  categories  are  then  differenced.  Appendix  B  shows  the  200  trials  and  the 
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raw  differenced  data.  The  differenced  data  distribution  allows  for  a  comparison  of  the 
macro  versus  micro  techniques.  A  histogram  of  this  distribution  is  shown  in  Figure  28. 


-0.05  -0.04  -0.03  -0.02  -0.01  0  .01 

Figure  28.  Differenced  Flyaway  vs.  Basic  Airframe  Data 

The  mean  of  the  distribution  is  -0.0208.  The  95%  confidence  interval  is  -0.0195 
to  -0.0222.  This  interval  does  not  contain  zero.  Therefore,  it  can  be  concluded  that 
statistically  there  is  a  difference  in  the  error  risk  between  estimating  at  the  Flyaway  level 
versus  the  Basic  Airframe  level. 

Bootstrap  Comparison  for  Flyaway  Versus  Other  Air  Vehicle 

The  bootstrap  technique  is  applied  in  an  identical  manner  for  the  Other  Air 
Vehicle  data  as  it  was  for  the  Flyaway  and  Basic  Airframe  components.  The  only 
difference  is  the  residuals  are  modeled  as  a  Normal  (0,  0.356)  distribution,  as  determined 
from  the  regression  equation.  The  frequency  chart  of  the  standard  deviation  from  the 
bootstrap  data  is  shown  in  Figure  29.  The  range  of  values  for  the  Other  Air  Vehicle 
component  is  wider  by  0.01  than  the  range  from  the  Flyaway  component.  While  the 
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explicit  values  of  this  range  are  different  from  the  Basic  Airframe,  it  is  of  interest  that  the 
spread  of  the  range  is  identical.  Again,  this  may  be  an  indication  that  there  is  a  difference 
in  the  error  components. 
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Figure  29.  Frequency  Chart  of  Other  Air  Vehicle  Bootstrap  Data 


The  resulting  200  standard  deviation  samples  from  the  Flyaway  data  and  Other 
Air  Vehicle  were  differenced.  See  Appendix  B  for  the  raw  data  and  differenced  data.  A 
histogram  of  the  differenced  data  is  shown  in  Figure  30. 


Figure  30.  Differenced  Flyaway  vs.  Other  Air  Vehicle 
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The  mean  of  the  distribution  is  -0.0445.  The  95%  confidence  interval  is  -0.0429 


to  -0.046.  This  interval  does  not  contain  zero.  Therefore,  it  can  be  concluded  that 
statistically  there  is  a  difference  in  the  error  risk  between  estimating  at  the  Flyaway  level 
versus  the  Other  Air  Vehicle  level. 

Multiple  Regression  Models  for  the  Inter-theatre  Airlift  Category 

Data 

The  data  for  the  Inter-theatre  Airlift  category  comes  from  two  sources,  the  Delta 
Research  Corporation  report  and  the  SPOs,  as  detailed  in  the  previous  chapter.  The  four 
aircraft  under  consideration  are  the  C-130,  C-141,  C-5,  and  C-17.  These  aircraft  were 
chosen  for  several  reasons.  First,  they  are  all  operational  aircraft  that  are  currently 
utilized  by  their  service.  Second,  there  are  multiple  data  points  available  from  which  to 
conduct  the  analysis.  Third,  the  characteristics  of  these  aircraft  provide  a  natural 
grouping  that  allows  for  a  homogeneous  database  to  analyze. 

Development  of  the  regression  model  for  the  Basic  Airframe,  Other  Air  Vehicle, 
and  Flyaway  cost  all  had  one  common  result.  The  parameter  estimates  for  the  C-141  data 
were  found  to  be  insignificant  in  all  of  the  models.  As  the  C-141  had  the  least  amount  of 
data  points,  this  is  not  a  major  limitation  and  the  C-141  data  was  discarded.  Therefore, 
all  three  regression  models  developed  below  use  only  C-130,  C-5,  and  C-17  data. 

Basic  Airframe  Model 

First,  the  dependent  variable  is  analyzed  to  ensure  it  is  continuous.  Figure  3 1 
shows  that  there  are  no  major  gaps  or  spikes  in  the  data.  This  leads  to  the  conclusion  that 
the  data  is  continuous. 
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Figure  31.  Continuous  Data  Histogram  for  Basic  Airframe 

The  next  step  is  to  determine  the  overall  predictive  capability  of  the  model.  As 
shown  in  Table  17,  the  p-value  is  <.0001  indicating  that  at  least  one  of  the  model 
parameters  has  predictive  capability. 


Table  17.  ANOVA  Table  for  Basic  Airframe 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Model 

4 

21.000082 

5.25002 

85.4912 

Error 

19 

1.166792 

0.06141 

Prob  >  F 

C.  Total 

23 

22.166874 

<.0001 

A  plot  of  Cook’s  D  is  the  diagnostic  for  influential  data  points.  As  shown  in 
Figure  32,  there  is  one  major  influential  data  point  above  the  0.5  threshold  level. 
Removing  that  data  point  and  re-running  the  model  resulted  in  two  other  influential  data 
points.  These  points  are  also  removed  from  the  model. 
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Figure  32.  Cook's  D  Influence  for  Basic  Airframe 


A  final  plot  of  Cook’s  D  Influence  (see  Figure  33)  shows  a  uniform  spread  of  the 
data,  with  each  value  below  0.25.  Thus,  the  final  model  does  not  include  influential  data 
points. 
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Figure  33.  Revised  Cook's  D  Influence  for  Basic  Airframe 


An  examination  of  the  Summary  of  Fit  table  (see  Table  18)  and  Parameter 
Estimates  shows  that  the  independent  variables  do  an  excellent  job  of  explaining  the 
variability  of  the  dependent  variable.  This  is  confirmed  by  the  R  value  of  0.947  and 
adjusted  R  value  of  0.936.  Additionally,  the  independent  variables  are  significant  with 
each  having  a  p-value  less  than  the  a  =  0.10  threshold.  The  VIF  scores  are  also  well 
below  the  threshold  of  ten,  verifying  that  multi-collinearity  does  not  exist. 
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Table  18.  Summary  of  Fit  and  Parameter  Estimates  for  Basic  Airframe 


RSquared 

0.947363 

RSquared  Adj 

0.936282 

Root  Mean  Square  Error 

0.247811 

Mean  of  Response 

3.7751 

Observations  (or  Sum  Wgts) 

24 

Summary  of  Fit 


Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

VIF 

Intercept 

4.5174542 

0.079756 

56.64 

<.0001 

Category  [11 

0.8095095 

0.101365 

7.99 

<.0001 

2.5655289 

Category[2] 

0.7282849 

0.145753 

5.00 

<.0001 

3.4449211 

Category[31 

-1.436094 

0.107786 

-13.32 

<.0001 

2.577618 

Cum  Qty 

-0.017515 

0.001677 

-10.44 

<.0001 

1.1791086 

Parameter  Estimates 


The  three  assumptions  of  normality,  independence,  and  constant  variance  must  be 
satisfied  before  accepting  the  model.  Figure  34  shows  a  plot  of  the  studentized  residuals 
to  test  for  normality.  The  visual  plot  exhibits  some  bell  shaped  characteristics  that  would 
be  expected  from  a  normal  distribution.  However,  the  dip  in  the  middle  leaves  some 
doubt.  For  a  more  robust  indicator,  the  Shapiro-Wilk  test  is  applied  and  a  p-value  of  0.84 
is  returned.  Thus,  the  assumption  of  nonnality  is  validated. 
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Figure  34.  Normality  Histogram  for  Basic  Airframe 


The  studentized  residuals  are  also  used  to  check  for  outliers.  As  shown  in  Table 
19,  the  furthest  value  from  zero  is  2.2 1 .  Therefore,  it  can  be  concluded  that  there  are  no 
outliers  in  this  model. 


Table  19.  Quartiles  for  Basic  Airframe 


100.0% 

maximum 

2.210 

99.5% 

2.210 

97.5% 

2.210 

90.0% 

1.282 

75.0% 

quartile 

0.895 

50.0% 

median 

-0.083 

25.0% 

quartile 

-0.886 

10.0% 

-1.414 

2.5% 

-1.681 

0.5% 

-1.681 

0.0% 

minimum 

-1.681 

The  autocorrelation  function  is  analyzed  to  check  for  independence.  As  shown  in 
Figure  35,  the  autocorrelation  plot  dies  down  quickly.  Also,  there  is  only  one  spike 
above  the  critical  lines.  The  assumption  of  independence  is  thus  met. 
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Figure  35.  Autocorrelation  Function  for  Basic  Airframe 

The  final  assumption  to  verify  is  constant  variance.  The  residual  by  predicted  plot 
(see  Figure  36)  is  used  as  the  descriptive  measure.  The  data  points  have  a  relatively  even 
spread,  but  a  potential  megaphone  pattern  is  present.  An  application  of  the  more  robust 
Breusch-Pagan  test  results  in  a  y2  statistic  of  0.783,  as  calculated  below. 


.01007851 

5 


1.166792  ~)2 
23  ) 


0.783 


2 

The  p-value  associated  with  the  %  statistic  is  0.94,  thereby  satisfying  the  assumption  of 
constant  variance. 
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Figure  36.  Residual  by  Predicted  for  Basic  Airframe 

The  Basic  Airframe  model  is  complete  and  has  passed  all  the  required 
diagnostics.  The  resulting  equation  is: 

Y=  4.517  +  0.809*CW(1)  +  0.728*C«t(2)  -  1.44*Cat(3)  -  QMl%*CumQty  +  z  (7) 

The  error  term  is  the  item  of  interest  to  perform  the  macro  versus  micro 
comparison.  The  underlying  distribution  of  the  residuals  is  shown  in  Figure  37.  The 
residuals  are  Normally  Distributed  with  p  =  0  and  a  =  0.225.  The  Kohnogorov-Smirnov 
Test  returns  a  p-value  of  >0. 15,  validating  the  normal  distribution  fit.  This  distribution  is 
critical  as  an  input  to  the  Bootstrap  technique  that  will  be  used  to  perform  the  macro 
versus  micro  comparison. 
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Figure  37.  Plot  of  Residuals  for  Basic  Airframe 

Other  Air  Vehicle  Model 

A  plot  of  the  dependent  variable  (see  Figure  38)  confirms  that  the  data  is 
continuous.  There  are  no  gaps  or  major  spikes  in  the  data. 
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Figure  38.  Continuous  Data  Histogram  for  Other  Air  Vehicle 

The  overall  predictive  capability  of  the  model  is  shown  in  the  ANOVA  Table  (see 
Table  20).  The  p-value  associated  with  the  F-statistic  is  <.0001.  This  is  lower  than  the 
a  =  0. 1  threshold.  From  this  it  can  be  concluded  that  at  least  one  of  the  parameters  in  the 
model  has  predictive  capability. 
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Table  20.  ANOVA  Table  for  Other  Air  Vehicle 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Model 

3 

8.895078 

2.96503 

34.0974 

Error 

17 

1.478279 

0.08696 

Prob  >  F 

C.  Total 

20 

10.373357 

<.0001 

The  plot  of  Cook’s  D  checks  for  influential  data  points  (see  Figure  39).  There  is 
one  data  point  above  the  0.50  threshold  level.  An  analysis  of  the  impact  of  this  data  point 
reveals  little  change  in  the  R  and  adjusted  R“  values.  Additionally,  the  significance  of 
neither  the  parameter  estimate  values  nor  their  associated  VIF  scores  changes 
dramatically  with  the  removal  of  the  data  point.  Therefore,  because  the  data  point’s  value 
of  0.506  is  close  to  the  threshold  value  and  it  does  not  appear  to  dramatically  change  the 
model,  it  remains  in  the  final  model. 


Figure  39.  Cook's  D  Influence  for  Other  Air  Vehicle 


The  Summary  of  Fit  table  and  Parameter  Estimates  (see  Table  21)  confirms  that 
the  model  does  a  good  job  of  explaining  the  variability  in  the  dependent  variable.  This  is 
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verified  through  the  R2  value  of  0.857  and  adjusted  R2  value  of  0.832.  All  of  the 
parameter  estimate  are  below  the  a  =  0. 10  threshold  except  for  the  cumulative  quantity 
parameter.  The  cumulative  quantity  parameter  serves  two  purposes:  to  capture  the 
learning  curve  effect,  and  to  represent  the  element  of  time  due  to  consecutive  production 
lot  buys.  An  analysis  of  the  raw  data  does  not  indicate  a  natural  decline  of  lot  cost  over 
time,  as  would  be  expected  from  a  learning  curve.  This  lack  of  learning  curve  effect  is  a 
likely  cause  of  the  high  p-value.  An  analysis  of  the  effects  of  removing  the  cumulative 
quantity  parameter  results  in  a  0.001  decrease  in  the  adjusted  R“  value.  Thus,  due  to  the 
dual  purpose  of  the  variable  and  the  fact  that  it  does  not  adversely  affect  the  outputs  of 
the  model,  it  remains  in  the  final  model. 

Table  21.  Summary  of  Fit  and  Parameter  Estimates  for  Other  Air  Vehicle 


RSquare 

0.857493 

RSquare  Adj 

0.832344 

Root  Mean  Square  Error 

0.294886 

Mean  of  Response 

2.706801 

Observations  (or  Sum  Wgts) 

21 

Summary  of  Fit 


Term 

Estimate 

Std  Error 

t  Ratio 

A 

O 

■— 

Cm 

VIF 

Intercept 

2.6529963 

0.10428 

25.44 

<.0001 

Category[l] 

-1.019602 

0.116718 

-8.74 

<.0001 

2.3723415 

Category[2] 

0.5212374 

0.157098 

3.32 

0.0041 

2.5948719 

Cum  Qty 

-0.002204 

0.002069 

-1.07 

0.3017 

1.146707 

Parameter  Estimates 


The  three  underlying  assumptions  are  tested  for  compliance  before  accepting  the 
model.  Figure  40  shows  a  histogram  of  the  studentized  residuals.  The  shape  of  the 
histogram  appears  to  be  bell-shaped.  The  Shapiro-Wilk  test  returns  a  p-value  of  0.3679. 
Thus,  the  assumption  of  nonnality  is  satisfied. 
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Figure  40.  Normality  Histogram  for  Other  Air  Vehicle 

Outliers  are  also  examined  with  the  studentized  residuals.  As  shown  in  Figure  40, 
all  the  data  points  are  between  negative  three  and  three.  Table  22  shows  the  quartiles  for 
the  Other  Air  Vehicle  model.  The  maximum  value  of  2.45  in  the  quartiles  table  confirms 
the  visual  diagnostic  that  outliers  are  not  present  in  the  model. 


Table  22.  Quartiles  for  Other  Air  Vehicle 


100.0% 

maximum 

2.450 

99.5% 

2.450 

97.5% 

2.450 

90.0% 

1.085 

75.0% 

quartile 

0.567 

50.0% 

median 

0.139 

25.0% 

quartile 

-0.602 

10.0% 

-1.689 

2.5% 

-2.227 

0.5% 

-2.227 

0.0% 

minimum 

-2.227 

The  autocorrelation  function  (see  Figure  41)  indicates  that  the  assumption  of 
independence  is  satisfied.  The  spikes  dampen  quickly  and  remain  within  the  critical 
values. 
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Lag 

AutoCorr 

0 

1.0000 

1 

0.7251 

2 

0.5069 

3 

0.4151 

4 

0.3419 

5 

0.2471 

6 

0.2068 

7 

0.1796 

8 

0.0663 

9 

-0.1284 

10 

-0.2073 

11 

-0.2563 

12 

-0.2497 

13 

-0.2329 

14 

-0.2014 

Figure  41.  Autocorrelation  Function  for  Other  Air  Vehicle 


The  residual  by  predicted  plot  (see  Figure  42)  has  a  uniform  spread  of  the  data 
around  zero.  This  indicates  constant  variance  of  the  residuals.  The  Breusch-Pagan  test 
returns  a  x  statistic  of  3.296. 

.06532 

4 

-  =  3.296 

f  1.4782  V 

l  21 _ j _ 

The  p-value  associated  with  this  test  statistic  is  0.348.  This  confirms  the  visual 
diagnostic  of  constant  variance. 
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Figure  42.  Residual  by  Predicted  Plot  for  Other  Air  Vehicle 

The  Other  Air  Vehicle  model  passes  all  the  necessary  diagnostics.  The  resulting 
equation  is: 

Y=  2.653  -  1.019*CW(1)  +  0.521*C«t(2)  -  0.0022*C«»i  Qty  +  £  (8) 

The  error  term  is  the  item  of  interest  to  perform  the  micro  versus  macro 
comparison.  The  underlying  distribution  of  the  residuals  is  shown  in  Figure  43.  The 
residuals  are  Normally  Distributed  with  a  p  =  0  and  a  =  0.272.  The  Kolmogorov- 
Smimov  Test  returns  a  p-value  of  >0. 15,  validating  the  normal  distribution  fit.  This 
distribution  is  the  critical  input  for  the  Bootstrap  technique. 
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Figure  43.  Plot  of  Residuals  for  Other  Air  Vehicle 

Flyaway  Cost  Model 

A  histogram  of  the  dependent  variable  (see  Figure  44)  reveals  that  there  are  no 
gaps  in  the  data.  Additionally,  no  bins  of  data  exist  that  would  indicate  a  split 
distribution.  Thus,  it  can  be  concluded  that  the  data  is  continuous. 


4 

i 

Figure  44.  Continuous  Data  Histogram  for  Flyaway  Cost 

The  ANOVA  table  (see  Table  23)  indicates  that  the  overall  model  has  predictive 
capability.  This  is  demonstrated  by  the  p-value  associated  with  the  F-statistic  of  <.0001. 
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Table  23.  ANOVA  Table  for  Flyaway  Cost 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Model 

4 

35.224151 

8.80604 

187.4490 

Error 

21 

0.986545 

0.04698 

Prob  >  F 

C.  Total 

25 

36.210695 

<.0001 

A  plot  of  Cook’s  D  found  two  influential  data  points  in  the  model.  These  points 
are  removed  and  Cook’s  D  is  re-analyzed.  Figure  45  shows  the  new  plot.  Although  all 
the  data  points  are  below  the  threshold  level,  there  is  one  data  point  that  is  separated  from 
the  rest.  An  analysis  of  this  data  point,  to  include  the  effect  on  the  adjusted  R“  values  and 
parameter  estimates,  verifies  that  this  point  is  not  influential. 
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Figure  45.  Cook's  D  Influence  for  Flyaway  Cost 


9 

The  independent  variables  explain  most  of  the  variability  in  the  model  with  a  R“ 
of  0.972  and  an  adjusted  R  of  0.967.  The  independent  variables  in  the  model  are  also 
significant  as  each  is  below  the  a  =  0.10  threshold.  Multi-collinearity  is  not  an  issue  as 
all  the  VIF  scores  are  below  ten.  These  results  are  shown  in  Table  24. 
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Table  24.  Summary  of  Fit  and  Parameter  Estimates  for  Flyaway  Cost 


RSquare 

0.972755 

RSquare  Adj 

0.967566 

Root  Mean  Square  Error 

0.216745 

Mean  of  Response 

4.598784 

Observations  (or  Sum  Wgts) 

26 

Summary  of  Fit 


Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

VIF 

Intercept 

4.7262224 

0.06626 

71.33 

<.0001 

Categoryfll 

0.3233277 

0.086936 

3.72 

0.0013 

2.5555149 

Category[2] 

0.6178742 

0.126029 

4.90 

<.0001 

3.5370266 

Category[31 

-1.867154 

0.089858 

-20.78 

<.0001 

2.7301896 

Cum  Qty 

-0.008976 

0.001117 

-8.03 

<.0001 

1.1097917 

Parameter  Estimates 


Before  accepting  the  model,  the  three  underlying  assumption  must  be  validated. 
Figure  46  shows  a  histogram  of  the  studentized  residuals.  The  data  shows  a  bell-shape 
that  indicates  a  normal  distribution.  This  is  confirmed  by  the  Shapiro-Wilk  p-value  of 
0.64.  Thus,  the  assumption  of  normality  is  satisfied. 


Figure  46.  Normality  Histogram  for  Flyaway  Cost 


Outliers  are  also  identified  by  the  studentized  residuals.  Figure  46  does  not  show 
any  outliers.  The  quartiles  table  confirms  this  visual  result  (see  Table  25).  The  furthest 
data  point  from  zero  is  2.59.  Therefore,  outliers  are  not  present  in  the  model. 
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Table  25.  Quartiles  for  Flyaway  Cost 


100.0% 

maximum 

2.599 

99.5% 

2.599 

97.5% 

2.599 

90.0% 

1.577 

75.0% 

quartile 

0.709 

50.0% 

median 

-0.013 

25.0% 

quartile 

-0.688 

10.0% 

-1.520 

2.5% 

-1.745 

0.5% 

-1.745 

0.0% 

minimum 

-1.745 

The  autocorrelation  function  (see  Figure  47)  confirms  independence  in  the  model. 
The  function  dampens  quickly  and  does  not  have  spikes  outside  the  critical  values. 
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0.2163 
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Figure  47.  Autocorrelation  Function  for  Flyaway  Cost 


The  last  diagnostic  is  constant  variance.  Figure  48  shows  a  plot  of  the  residuals 
by  predicted.  This  visual  plot  shows  a  uniform  spread  around  zero,  indicating  constant 

2 

variance.  The  Breusch-Pagan  test  quantitatively  confirms  this  visual  conclusion.  The  % 
statistic  value  is  2.931. 
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The  p-value  associated  with  the  %2  statistic  is  0.57.  Thus,  the  assumption  of 
constant  variance  is  validated. 
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Figure  48.  Residual  by  Predicted  Plot  for  Flyaway  Cost 

The  Flyaway  Cost  model  is  complete  and  has  passed  all  the  diagnostics.  The 
resultant  equation  is: 

Y=  4.726  +  0.323*CW(1)  +  0.618*C«t(2)  -  1.867*C«t(3)  -  0M89*CumQty  +  £  (9) 

The  error  term  is  the  required  input  for  the  Bootstrap  technique  to  perform  the 
micro  versus  macro  comparison.  Figure  49  shows  the  resulting  distribution  of  the 
residuals.  The  residuals  are  Normally  Distributed  with  a  p  =  0  and  a  =  0.199.  The 
Kolmogorov-Smirnov  returns  a  p-value  of  >0. 15,  validating  the  normal  distribution  fit. 
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Figure  49.  Plot  of  Residuals  for  Flyaway  Cost 


Bootstrap  Comparison  for  Flyaway  Versus  Basic  Airframe 

Fixed  X,  residual  re-sampling  is  the  next  step  in  the  analysis.  Starting  with  the 
Flyaway  Cost  regression  equation,  Crystal  Ball  performs  the  Bootstrap  technique.  This  is 
accomplished  by  defining  the  assumption  cell  as  a  Normal  (0,  0.119)  Distribution.  This 
distribution  represents  the  residuals  as  determined  from  the  regression  equation.  200 
bootstrap  samples  are  generated,  with  1000  iterations  occurring  within  each  sample.  The 
frequency  histogram  for  the  standard  deviation  from  these  trials  is  shown  in  Figure  50. 

As  anticipated,  the  majority  of  the  trials  are  around  the  0.20  value,  with  a  relatively  tight 
range  from  0.19  to  0.21. 
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Figure  50.  Frequency  Chart  of  Flyaway  Bootstrap  Data 


The  process  is  repeated  for  the  Basic  Airframe  data.  The  primary  change  in  the 
application  of  the  Bootstrap  technique  is  the  residuals  are  modeled  as  a  Normal  (0,  0.225) 
distribution,  as  determined  from  the  regression  equation.  The  resulting  Bootstrap 
standard  deviation  frequency  chart  is  shown  below  (see  Figure  51).  One  characteristic 
that  is  noticeably  different  from  the  Flyaway  Cost  frequency  chart  is  the  spread  of  the 
data.  The  Basic  Airframe  has  a  wider  spread,  by  0.01.  This  may  be  a  significant 
indicator  of  a  difference  between  estimating  the  two  error  components. 
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200  Trials 


Forecast:  Standard  Deviation 
Frequency  Chart 
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Figure  51.  Frequency  Chart  of  Basic  Airframe  Bootstrap  Data 


The  standard  error  of  the  200  Bootstrap  samples  from  the  Flyaway  cost  and  Basic 
Airframe  are  paired  and  differenced.  Appendix  C  shows  the  200  trials  and  differenced 
data.  The  resulting  distribution  is  the  critical  component  to  perfonn  the  macro  versus 
micro  comparison.  A  histogram  of  this  distribution  is  shown  in  Figure  52. 


Figure  52.  Differenced  Flyaway  vs.  Basic  Airframe  Data 


The  mean  of  the  distribution  is  -0.027.  The  95%  confidence  interval  is  -0.026  to 
-0.028.  This  interval  does  not  contain  zero.  Therefore,  it  can  be  concluded  that  there  is  a 
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statistical  difference  in  the  error  risk  between  estimating  at  the  Flyaway  level  versus  the 
Basic  Airframe  level. 


Bootstrap  Comparison  for  Flyaway  Versus  Other  Air  Vehicle 

The  Bootstrap  technique  is  applied  in  an  identical  manner  for  the  Other  Air 
Vehicle  data  as  it  was  for  the  Flyaway  and  Basic  Airframe  components.  The  only 
difference  is  the  residuals  are  modeled  as  a  Normal  (0,  0.272)  Distribution,  as  determined 
from  the  regression  equation.  The  frequency  chart  of  the  standard  deviation  from  the 
Bootstrap  data  is  shown  in  Figure  53.  The  range  of  values  for  the  Other  Air  Vehicle 
component  is  0.26  to  0.29,  which  is  wider  by  0.01  than  the  range  from  the  Flyaway 
component.  This  may  be  an  indication  that  there  is  a  difference  in  the  error  components. 
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Figure  53.  Frequency  Chart  of  Other  Air  Vehicle  Bootstrap  Data 


The  resulting  200  standard  deviation  samples  from  the  Flyaway  and  Other  Air 
Vehicle  data  are  differenced.  See  Appendix  C  for  the  raw  and  differenced  data.  A 
histogram  of  the  differenced  data  is  shown  in  Figure  54. 
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Figure  54.  Differenced  Flyaway  vs.  Other  Air  Vehicle  Data 

The  mean  of  the  distribution  is  -0.0732.  The  95%  confidence  interval  is  -0.0722 
to  -0.0742.  This  interval  does  not  contain  zero.  Therefore,  it  can  be  concluded  that  there 
is  a  statistical  difference  in  the  error  risk  between  estimating  at  the  Flyaway  level  versus 
the  Other  Air  Vehicle  level. 

Summary 

This  chapter  has  shown  the  analysis  of  the  data  required  to  perform  the  micro 
versus  macro  comparison.  Regression  equations  and  Bootstrap  samples  were  developed 
for  both  categories  of  data.  This  data  was  then  differenced  and  paired  to  form  a 
distribution.  The  four  resulting  distributions  (see  Figure  55)  are  the  basis  for  the 
conclusions  drawn  in  Chapter  5. 
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Figure  55.  Distributions  from  Macro  vs.  Micro  Comparisons 
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V.  Conclusions 


Importance  of  Findings 

This  research  is  important  for  several  reasons.  First,  the  cost  analysis  career  field 
is  shrinking.  As  demonstrated  by  the  ASC  example  in  Chapter  1,  there  has  been  a 
dramatic  reduction  in  cost  authorizations  over  the  past  decade.  Cost  analysts  are 
therefore  becoming  a  scarce  resource.  When  confronted  with  the  challenge  of  developing 
a  cost  estimate,  program  managers  need  to  know  how  to  optimize  this  resource.  By 
understanding  the  advantages  and  disadvantages  from  an  estimation  error  risk  perspective 
of  estimating  at  differing  WBS  level,  optimal  allocation  of  cost  analysis  resources  can  be 
achieved.  Second,  in  order  to  achieve  cycle  time  reduction  goals,  the  time  to  develop  a 
cost  estimate  is  compressed.  As  a  result,  cost  estimates  need  to  be  developed  more 
quickly  while  still  maintaining  a  satisfactory  level  of  fidelity.  This  lends  to  the 
conclusion  that  using  the  time  consuming  grass-roots  techniques  will  not  be  possible. 
Rather,  estimation  will  occur  at  the  highest  WBS  level  possible,  while  still  achieving  a 
satisfactory  level  of  confidence  in  the  estimate.  This  research  provides  the  analysis 
necessary  to  understand  the  tradeoffs  implicit  in  estimating  at  the  differing  WBS  levels. 
When  making  resource  allocation  decisions  under  a  constrained  environment,  program 
managers  can  then  apply  this  information. 

Limitations 

There  are  several  limitations  to  this  research.  First,  only  recurring  data  is 
considered  in  the  analysis.  The  estimation  error  risk  of  non-recurring  data  is  not 
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considered.  Second,  the  weapons  systems  analyzed  are  limited  to  aircraft  systems, 
specifically  fighters  and  inter-theatre  airlift  aircraft.  To  extrapolate  the  results  of  the 
analysis  to  data  outside  of  aircraft  weapons  systems  is  inappropriate.  Likewise,  to 
extrapolate  the  results  to  other  categories  of  aircraft,  such  as  bombers  is  inappropriate. 
Third,  the  WBS  level  comparison  is  limited  to  level  one  versus  level  two.  Conclusions 
about  lower  WBS  levels  are  not  considered.  Finally,  the  WBS  level  two  breakdown  is 
not  a  pure  MIL-HDBK-881  breakout.  Conclusions  can  only  be  drawn  about  a  level  one 
versus  level  two  comparison  with  regards  to  the  breakout  of  WBS  level  two  into  the 
Basic  Airframe  and  Other  Air  Vehicle  components. 

Discussion  of  Results 

Starting  with  the  Fighter  category,  there  is  a  statistically  significant  difference 
between  estimating  at  the  Flyaway  Cost  level  versus  the  Basic  Airframe  and  the  Other 
Air  Vehicle  level.  This  is  confirmed  by  the  95%  confidence  intervals  around  the  mean  of 
the  differenced  distribution,  which  do  not  contain  zero  for  either  model  comparison.  For 
the  Flyaway  Cost  versus  Basic  Airframe  model,  the  mean  of  the  distribution  is  -0.0208 
with  a  95%  confidence  interval  of  (-0.0195,  -0.0222).  For  the  Flyaway  Cost  versus  Other 
Air  Vehicle  model,  the  mean  is  -0.0445  with  a  95%  confidence  interval  of  (-0.0429, 
-0.046).  Several  additional  conclusions  can  be  drawn  from  this.  First,  there  is  clearly 
more  error  risk  in  the  estimation  of  the  Other  Air  Vehicle  model  than  the  Basic  Airframe 
model.  This  indicates  that  program  managers  should  allocate  more  time  and  resource  to 
the  development  of  the  Other  Air  Vehicle  estimate  than  to  the  Basic  Airframe  estimate  if 
the  estimate  is  being  developed  at  WBS  level  two.  The  second  conclusion  was  one  not 
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anticipated  when  this  research  began.  The  differenced  distributions  are  calculated  by 
subtracting  the  WBS  level  two  data  from  the  WBS  level  one  data.  As  shown  above,  the 
mean  and  resultant  95%  confidence  intervals  of  both  these  distributions  are  negative. 

This  leads  to  the  conclusion  that  estimating  at  WBS  level  one  has  less  error  risk  than 
estimating  at  WBS  level  two.  There  are  several  possible  reasons  for  this.  It  could  be  that 
when  estimating  at  the  lower  levels,  the  details  of  the  estimate  cloud  the  bigger  picture, 
leading  to  inaccurate  or  inappropriate  model  inputs  from  experts.  In  other  words,  it  may 
be  harder  to  breakdown  an  estimate  to  the  individual  components  without  adding 
additional  error.  Another  possible  explanation  is  that  the  positive  and  negative  error  risks 
in  the  individual  components  cancel  each  other  out  as  they  accumulate  at  higher  levels. 
Although  this  research  cannot  conclude  with  any  certainty  why  the  WBS  level  one  error 
risk  is  less  than  the  WBS  level  two  error  risk,  the  above  possibilities  are  reasonable 
explanations. 

The  results  from  the  Inter-theatre  Airlift  category  are  similar.  There  is  a 
statistically  significant  difference  in  the  estimating  error  between  estimating  at  WBS 
level  one  and  level  two.  The  mean  of  the  distribution  for  the  Flyaway  Cost  versus  Basic 
Airframe  is  -0.027  with  a  95%  confidence  interval  of  (-0.026,  -0.028).  The  mean  of  the 
distribution  for  the  Flyaway  Cost  versus  Other  Air  Vehicle  is  -0.0732  with  a  95% 
confidence  interval  of  (-0.0722,  -0.0742).  As  neither  confidence  interval  encompasses 
zero,  it  is  appropriate  to  say  that  there  is  a  statistical  difference  between  the  two.  Like  the 
Fighter  category  conclusions,  there  is  more  estimation  error  risk  in  the  Other  Air  Vehicle 
model  than  the  Basic  Airframe.  This  indicates  that  program  managers  should  allocate 
more  resources  to  the  Other  Air  Vehicle  portion  of  their  estimates.  Also,  as  with  the 
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Fighter  category  results,  it  is  determined  that  there  is  more  estimation  error  risk  when 
estimating  at  WBS  level  two  than  at  WBS  level  one.  The  same  rationale  explained  for 
the  Fighter  category  is  applicable  to  the  Inter- theatre  Airlift  results. 

Practical  versus  Statistical  Significance 

Despite  the  conclusions  above  regarding  the  statistically  significant  differences 
between  estimating  at  the  varying  WBS  levels,  there  is  a  practical  application  perspective 
to  consider.  The  estimation  errors  from  the  models  are  extremely  small  considering  the 
multi-million  dollar  costs  of  aircraft  weapon  systems.  Quantitatively,  the  dollar  amount 
differences  are  shown  in  Table  26. 


Table  26.  Practical  Significance  of  WBS  Estimation  Levels 


Category 

Fighter 

Inter-theatre  Airlift 

Flyaway  vs.  Basic  Airframe 

$44,137.60 

$68,682.90 

Flyaway  vs.  Other  Air  Vehicle 

$124,097.40 

$157,584.96 

These  dollar  amounts  are  so  small  that  although  there  is  a  statistically  significant 
difference,  there  is  little  difference  from  a  practical  standpoint.  In  most  cases,  the  error 
risk  is  simply  not  large  enough  for  program  managers  to  be  overly  concerned  when 
allocating  resources.  As  a  result,  it  is  anticipated  that  program  managers  will  allocate 
resources  based  on  other  considerations,  such  as  time  constraints  or  desired  level  of 
visibility  into  the  estimate. 

Future  Research 

There  are  several  areas  related  to  the  methodology  of  this  research  that  can  be 
explored  in  future  research.  First,  an  examination  of  the  non-recurring  estimating  error 
between  differing  WBS  levels  could  be  examined.  This  is  a  natural  extension  of  the 
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recurring  estimation  error  analyzed  in  this  research.  Second,  a  comparison  of  the 
estimation  error  difference  at  WBS  level  two  versus  WBS  level  three  could  be  explored. 
Although  other  variations  of  WBS  level  comparisons  could  be  made,  a  level  two  versus 
level  three  would  be  most  useful  to  the  practitioner.  Third,  this  methodology  could  be 
applied  to  different  weapon  systems  than  aircraft.  These  future  research  areas  would  be  a 
natural  bridge  to  the  limitations  described  above. 
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Appendix  A.  Basic  Airframe  Regression  Model  at  Quantity  100 


Each  aircraft’s  cost  is  calculated  at  quantity  100.  The  Delta  Research  Corporation 
database  includes  the  learning  curve  percentage  and  quantity  100  costs  calculated  for  the 
four  aircraft  under  consideration.  Under  this  approach  all  of  the  physical,  performance, 
and  technical  independent  variables  are  able  to  be  considered  in  the  regression  model. 
Due  to  the  fact  that  only  four  data  points  are  used  for  the  model,  a  maximum  of  only  two 
parameters  can  be  included  in  the  model.  Through  a  multitude  of  regression  analysis 
runs,  two  variables  are  found  to  be  significant:  Airframe  Unit  Weight  (AUW),  and  Max 
Gross  Take-Off  Weight.  This  result  is  shown  below. 


Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

Intercept 

2.4681831 

0.06634 

37.20 

0.0171 

Airframe  Unit  Weight 

0.001279 

0.00012 

10.62 

0.0598 

Max  Gross  TO  Weight 

-0.00039 

0.000037 

-10.47 

0.0606 

However,  the  VIF  scores  for  the  two  parameters  are  greater  than  3 1,000.  The 
multivariate  plot  below  confirms  this  multi-collinearity  problem. 


102 


Due  to  the  multi-collinearity  problem  with  the  two  parameters,  a  new  model  must 
be  developed.  The  results  of  this  analysis  lead  to  a  one-parameter  model.  Airframe  unit 
weight  is  selected  as  the  independent  variable.  The  selection  of  AUW  as  the  independent 
variable  is  consistent  with  the  RAND  studies.  The  ANOVA  output  for  this  model  is 
shown  below. 


RSquare 

0.871481 

RSquare  Adj 

0.807221 

Root  Mean  Square  Error 

0.584932 

Mean  of  Response 

2.916314 

Observations  (or  Sum  Wgts) 

4 

Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

Intercept 

2.0250211 

0.379622 

5.33 

0.0334 

Airframe  Unit 

Weight 

0.0000187 

0.000005 

3.68 

0.0665 
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Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Model 

1 

4.6401337 

4.64013 

13.5619 

Error 

2 

0.6842920 

0.34215 

Prob  >  F 

C.  Total 

3 

5.3244257 

0.0665 

The  R2  value  is  high  at  0.87.  Additionally,  the  independent  variable  is  significant. 
However,  a  plot  of  Cook’s  D  Influence  shows  that  the  B-l  data  point  is  highly  influential. 
Because  the  B-l  is  such  a  highly  influential  data  point,  the  result  above  cannot  be 
considered  accurate.  This  problem,  plus  the  fact  that  the  original  methodology  results  in 
a  more  parsimonious  model,  is  the  primary  reason  this  methodology  is  not  selected. 


Appendix  B:  Bootstrap  Data  for  Fighter  Category 


Bootstrap 

Sample 

Flyaway 
Std.  Dev. 

Basic 
Airframe 
Std.  Dev. 

Differenced 

Distribution 

Flyaway 
Std.  Dev. 

Other  Air 
Vehicle 
Std.  Dev. 

Differenced 

Distribution 

1 

0.31200 

0.34725 

-0.03525 

0.31200 

0.36313 

-0.05113 

2 

0.31691 

0.33221 

-0.01530 

0.31691 

0.35786 

-0.04095 

3 

0.30899 

0.32262 

-0.01363 

0.30899 

0.38152 

-0.07253 

4 

0.32144 

0.35048 

-0.02904 

0.32144 

0.36832 

-0.04688 

5 

0.31060 

0.33852 

-0.02792 

0.31060 

0.36794 

-0.05733 

6 

0.32390 

0.33068 

-0.00678 

0.32390 

0.37039 

-0.04649 

7 

0.32115 

0.33535 

-0.01420 

0.32115 

0.36314 

-0.04199 

8 

0.32144 

0.34780 

-0.02636 

0.32144 

0.36734 

-0.04590 

9 

0.31761 

0.33342 

-0.01582 

0.31761 

0.36945 

-0.05185 

10 

0.33090 

0.33719 

-0.00629 

0.33090 

0.35523 

-0.02434 

11 

0.30494 

0.34121 

-0.03627 

0.30494 

0.35503 

-0.05009 

12 

0.32175 

0.33623 

-0.01448 

0.32175 

0.35915 

-0.03740 

13 

0.32275 

0.34568 

-0.02293 

0.32275 

0.37205 

-0.04930 

14 

0.30876 

0.34927 

-0.04052 

0.30876 

0.36581 

-0.05705 

15 

0.31089 

0.34404 

-0.03314 

0.31089 

0.37517 

-0.06427 

16 

0.31643 

0.33874 

-0.02230 

0.31643 

0.35587 

-0.03943 

17 

0.30911 

0.33771 

-0.02860 

0.30911 

0.35914 

-0.05003 

18 

0.31497 

0.33389 

-0.01892 

0.31497 

0.36791 

-0.05294 

19 

0.31639 

0.33421 

-0.01783 

0.31639 

0.37894 

-0.06255 

20 

0.30953 

0.34693 

-0.03741 

0.30953 

0.36575 

-0.05622 

21 

0.31620 

0.32341 

-0.00721 

0.31620 

0.37226 

-0.05607 

22 

0.31514 

0.34193 

-0.02679 

0.31514 

0.35051 

-0.03537 

23 

0.30959 

0.34085 

-0.03125 

0.30959 

0.35214 

-0.04255 

24 

0.31481 

0.34234 

-0.02753 

0.31481 

0.35565 

-0.04084 

25 

0.32020 

0.33532 

-0.01512 

0.32020 

0.35418 

-0.03398 

26 

0.31022 

0.33156 

-0.02133 

0.31022 

0.36775 

-0.05753 

27 

0.31440 

0.34376 

-0.02936 

0.31440 

0.37363 

-0.05923 

28 

0.31226 

0.33411 

-0.02184 

0.31226 

0.36216 

-0.04989 

29 

0.30466 

0.32629 

-0.02163 

0.30466 

0.36154 

-0.05688 

30 

0.31337 

0.34028 

-0.02691 

0.31337 

0.36413 

-0.05076 

31 

0.30748 

0.33379 

-0.02631 

0.30748 

0.35938 

-0.05190 

32 

0.31895 

0.34812 

-0.02917 

0.31895 

0.35751 

-0.03856 

33 

0.32049 

0.32818 

-0.00769 

0.32049 

0.36510 

-0.04461 

34 

0.31276 

0.33671 

-0.02396 

0.31276 

0.35511 

-0.04235 

35 

0.32324 

0.33999 

-0.01674 

0.32324 

0.35549 

-0.03224 

36 

0.32502 

0.35018 

-0.02516 

0.32502 

0.35225 

-0.02723 

37 

0.31024 

0.32606 

-0.01582 

0.31024 

0.35423 

-0.04399 

38 

0.31650 

0.34334 

-0.02684 

0.31650 

0.35091 

-0.03441 

39 

0.31673 

0.32940 

-0.01267 

0.31673 

0.36816 

-0.05144 
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Bootstrap 

Sample 

Flyaway 
Std.  Dev. 

Basic 
Airframe 
Std.  Dev. 

Differenced 

Distribution 

Flyaway 
Std.  Dev. 

Other  Air 
Vehicle 
Std.  Dev. 

Differenced 

Distribution 

40 

0.32122 

0.32633 

-0.00511 

0.32122 

0.35948 

-0.03826 

41 

0.31769 

0.35776 

-0.04008 

0.31769 

0.36256 

-0.04487 

42 

0.31995 

0.34330 

-0.02334 

0.31995 

0.36664 

-0.04669 

43 

0.29852 

0.33740 

-0.03888 

0.29852 

0.37376 

-0.07524 

44 

0.31292 

0.34310 

-0.03018 

0.31292 

0.35478 

-0.04186 

45 

0.32016 

0.34913 

-0.02896 

0.32016 

0.36492 

-0.04476 

46 

0.30334 

0.32906 

-0.02572 

0.30334 

0.35748 

-0.05414 

47 

0.32800 

0.32296 

0.00504 

0.32800 

0.36038 

-0.03237 

48 

0.31997 

0.35307 

-0.03310 

0.31997 

0.35525 

-0.03528 

49 

0.33155 

0.32845 

0.00310 

0.33155 

0.36164 

-0.03010 

50 

0.32328 

0.33848 

-0.01520 

0.32328 

0.36489 

-0.04161 

51 

0.31389 

0.33485 

-0.02096 

0.31389 

0.34483 

-0.03094 

52 

0.32569 

0.34792 

-0.02223 

0.32569 

0.36104 

-0.03535 

53 

0.31654 

0.33669 

-0.02015 

0.31654 

0.36494 

-0.04840 

54 

0.32609 

0.34379 

-0.01770 

0.32609 

0.36919 

-0.04310 

55 

0.31557 

0.34124 

-0.02567 

0.31557 

0.35355 

-0.03798 

56 

0.32774 

0.33493 

-0.00720 

0.32774 

0.36090 

-0.03316 

57 

0.32098 

0.34480 

-0.02383 

0.32098 

0.36833 

-0.04735 

58 

0.31970 

0.33828 

-0.01858 

0.31970 

0.37385 

-0.05415 

59 

0.31184 

0.32851 

-0.01666 

0.31184 

0.35632 

-0.04447 

60 

0.30860 

0.33486 

-0.02625 

0.30860 

0.35470 

-0.04610 

61 

0.32152 

0.33658 

-0.01506 

0.32152 

0.35300 

-0.03148 

62 

0.31312 

0.33150 

-0.01839 

0.31312 

0.34268 

-0.02956 

63 

0.31799 

0.33669 

-0.01870 

0.31799 

0.36650 

-0.04851 

64 

0.30535 

0.32727 

-0.02192 

0.30535 

0.35430 

-0.04894 

65 

0.32738 

0.33648 

-0.00909 

0.32738 

0.35810 

-0.03072 

66 

0.31790 

0.33688 

-0.01897 

0.31790 

0.35134 

-0.03344 

67 

0.31679 

0.34501 

-0.02822 

0.31679 

0.35999 

-0.04320 

68 

0.33066 

0.33598 

-0.00532 

0.33066 

0.35576 

-0.02510 

69 

0.29733 

0.34369 

-0.04636 

0.29733 

0.35739 

-0.06005 

70 

0.32770 

0.33981 

-0.01210 

0.32770 

0.35280 

-0.02510 

71 

0.32092 

0.33439 

-0.01347 

0.32092 

0.35076 

-0.02984 

72 

0.31177 

0.34487 

-0.03310 

0.31177 

0.35580 

-0.04403 

73 

0.30838 

0.33953 

-0.03114 

0.30838 

0.35389 

-0.04551 

74 

0.31052 

0.33034 

-0.01982 

0.31052 

0.36200 

-0.05148 

75 

0.30705 

0.33448 

-0.02743 

0.30705 

0.36828 

-0.06123 

76 

0.32065 

0.33703 

-0.01638 

0.32065 

0.35407 

-0.03341 

77 

0.31785 

0.33104 

-0.01319 

0.31785 

0.36992 

-0.05207 

78 

0.31782 

0.33658 

-0.01876 

0.31782 

0.35710 

-0.03928 

79 

0.30985 

0.33846 

-0.02861 

0.30985 

0.35061 

-0.04077 

80 

0.30855 

0.34161 

-0.03306 

0.30855 

0.35946 

-0.05092 

81 

0.31402 

0.34283 

-0.02881 

0.31402 

0.35643 

-0.04241 
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Bootstrap 

Sample 

Flyaway 
Std.  Dev. 

Basic 
Airframe 
Std.  Dev. 

Differenced 

Distribution 

Flyaway 
Std.  Dev. 

Other  Air 
Vehicle 
Std.  Dev. 

Differenced 

Distribution 

82 

0.32530 

0.33825 

-0.01295 

0.32530 

0.35975 

-0.03446 

83 

0.31430 

0.33854 

-0.02424 

0.31430 

0.38047 

-0.06617 

84 

0.31819 

0.33226 

-0.01407 

0.31819 

0.36775 

-0.04956 

85 

0.32115 

0.33356 

-0.01240 

0.32115 

0.36294 

-0.04179 

86 

0.31805 

0.34462 

-0.02658 

0.31805 

0.35965 

-0.04161 

87 

0.32785 

0.34047 

-0.01262 

0.32785 

0.35567 

-0.02783 

88 

0.31280 

0.33944 

-0.02664 

0.31280 

0.36332 

-0.05053 

89 

0.31895 

0.34130 

-0.02234 

0.31895 

0.35703 

-0.03808 

90 

0.32154 

0.33789 

-0.01634 

0.32154 

0.37491 

-0.05336 

91 

0.31921 

0.32578 

-0.00657 

0.31921 

0.36347 

-0.04426 

92 

0.31657 

0.33497 

-0.01841 

0.31657 

0.35339 

-0.03683 

93 

0.31643 

0.35217 

-0.03574 

0.31643 

0.35657 

-0.04013 

94 

0.32654 

0.33996 

-0.01342 

0.32654 

0.36079 

-0.03425 

95 

0.32764 

0.33910 

-0.01146 

0.32764 

0.35942 

-0.03178 

96 

0.31822 

0.33753 

-0.01931 

0.31822 

0.37284 

-0.05462 

97 

0.31599 

0.33884 

-0.02284 

0.31599 

0.35010 

-0.03411 

98 

0.31138 

0.33393 

-0.02255 

0.31138 

0.37135 

-0.05997 

99 

0.30675 

0.33189 

-0.02514 

0.30675 

0.35724 

-0.05048 

100 

0.31774 

0.32709 

-0.00935 

0.31774 

0.36143 

-0.04369 

101 

0.31392 

0.33523 

-0.02131 

0.31392 

0.35599 

-0.04207 

102 

0.32351 

0.33759 

-0.01407 

0.32351 

0.35383 

-0.03031 

103 

0.32326 

0.34224 

-0.01898 

0.32326 

0.37019 

-0.04693 

104 

0.31237 

0.34182 

-0.02945 

0.31237 

0.36268 

-0.05031 

105 

0.32726 

0.34258 

-0.01533 

0.32726 

0.36654 

-0.03928 

106 

0.31281 

0.33436 

-0.02155 

0.31281 

0.36340 

-0.05059 

107 

0.31970 

0.33671 

-0.01701 

0.31970 

0.36018 

-0.04048 

108 

0.31700 

0.34109 

-0.02409 

0.31700 

0.36445 

-0.04745 

109 

0.31536 

0.35052 

-0.03515 

0.31536 

0.34669 

-0.03133 

110 

0.31354 

0.32475 

-0.01121 

0.31354 

0.37750 

-0.06396 

111 

0.32650 

0.32965 

-0.00315 

0.32650 

0.34893 

-0.02243 

112 

0.31230 

0.33212 

-0.01981 

0.31230 

0.36230 

-0.05000 

113 

0.32620 

0.34032 

-0.01412 

0.32620 

0.34926 

-0.02306 

114 

0.30902 

0.34564 

-0.03663 

0.30902 

0.35209 

-0.04308 

115 

0.31358 

0.34014 

-0.02656 

0.31358 

0.36786 

-0.05428 

116 

0.32298 

0.32597 

-0.00299 

0.32298 

0.36827 

-0.04528 

117 

0.30955 

0.33630 

-0.02675 

0.30955 

0.34791 

-0.03836 

118 

0.32674 

0.33990 

-0.01316 

0.32674 

0.37309 

-0.04635 

119 

0.33086 

0.33600 

-0.00514 

0.33086 

0.36378 

-0.03292 

120 

0.32873 

0.33146 

-0.00273 

0.32873 

0.36933 

-0.04060 

121 

0.31670 

0.32662 

-0.00992 

0.31670 

0.36237 

-0.04567 

122 

0.31083 

0.33692 

-0.02610 

0.31083 

0.36045 

-0.04963 

123 

0.31358 

0.32608 

-0.01250 

0.31358 

0.34145 

-0.02787 
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Bootstrap 

Sample 

Flyaway 
Std.  Dev. 

Basic 
Airframe 
Std.  Dev. 

Differenced 

Distribution 

Flyaway 
Std.  Dev. 

Other  Air 
Vehicle 
Std.  Dev. 

Differenced 

Distribution 

124 

0.31008 

0.33485 

-0.02477 

0.31008 

0.35525 

-0.04517 

125 

0.31006 

0.33571 

-0.02565 

0.31006 

0.36191 

-0.05185 

126 

0.30724 

0.33807 

-0.03082 

0.30724 

0.37023 

-0.06299 

127 

0.30356 

0.33367 

-0.03011 

0.30356 

0.36783 

-0.06427 

128 

0.30193 

0.33541 

-0.03347 

0.30193 

0.36094 

-0.05900 

129 

0.30953 

0.35377 

-0.04424 

0.30953 

0.37181 

-0.06228 

130 

0.31707 

0.33449 

-0.01743 

0.31707 

0.36180 

-0.04473 

131 

0.33137 

0.33939 

-0.00803 

0.33137 

0.35478 

-0.02341 

132 

0.32301 

0.33363 

-0.01062 

0.32301 

0.36100 

-0.03799 

133 

0.31911 

0.32650 

-0.00739 

0.31911 

0.35954 

-0.04043 

134 

0.31803 

0.32735 

-0.00932 

0.31803 

0.36048 

-0.04245 

135 

0.30785 

0.33431 

-0.02646 

0.30785 

0.36444 

-0.05659 

136 

0.30344 

0.32326 

-0.01982 

0.30344 

0.36723 

-0.06379 

137 

0.30994 

0.33582 

-0.02588 

0.30994 

0.36526 

-0.05532 

138 

0.31387 

0.33413 

-0.02026 

0.31387 

0.36862 

-0.05474 

139 

0.32242 

0.34147 

-0.01905 

0.32242 

0.36427 

-0.04185 

140 

0.32036 

0.35839 

-0.03803 

0.32036 

0.35496 

-0.03460 

141 

0.31301 

0.34916 

-0.03614 

0.31301 

0.35479 

-0.04178 

142 

0.32124 

0.32798 

-0.00674 

0.32124 

0.36852 

-0.04728 

143 

0.30961 

0.34433 

-0.03472 

0.30961 

0.36825 

-0.05865 

144 

0.31667 

0.33274 

-0.01606 

0.31667 

0.35759 

-0.04092 

145 

0.30951 

0.33741 

-0.02790 

0.30951 

0.36901 

-0.05950 

146 

0.30075 

0.33404 

-0.03329 

0.30075 

0.36498 

-0.06423 

147 

0.30963 

0.33841 

-0.02878 

0.30963 

0.34470 

-0.03507 

148 

0.31433 

0.33400 

-0.01967 

0.31433 

0.36110 

-0.04677 

149 

0.31682 

0.34762 

-0.03079 

0.31682 

0.35873 

-0.04190 

150 

0.31871 

0.33686 

-0.01815 

0.31871 

0.35951 

-0.04080 

151 

0.30887 

0.33165 

-0.02278 

0.30887 

0.37641 

-0.06754 

152 

0.31205 

0.34679 

-0.03474 

0.31205 

0.35137 

-0.03933 

153 

0.32351 

0.35072 

-0.02721 

0.32351 

0.37138 

-0.04787 

154 

0.32136 

0.33686 

-0.01550 

0.32136 

0.35970 

-0.03834 

155 

0.30833 

0.34728 

-0.03895 

0.30833 

0.37671 

-0.06838 

156 

0.31694 

0.34892 

-0.03198 

0.31694 

0.37336 

-0.05642 

157 

0.31752 

0.33471 

-0.01720 

0.31752 

0.36546 

-0.04795 

158 

0.32313 

0.32956 

-0.00643 

0.32313 

0.34931 

-0.02618 

159 

0.30526 

0.32490 

-0.01964 

0.30526 

0.34441 

-0.03915 

160 

0.32692 

0.33807 

-0.01115 

0.32692 

0.34624 

-0.01933 

161 

0.31750 

0.33726 

-0.01975 

0.31750 

0.36832 

-0.05082 

162 

0.31244 

0.33477 

-0.02233 

0.31244 

0.35993 

-0.04749 

163 

0.32330 

0.34273 

-0.01943 

0.32330 

0.35956 

-0.03626 

164 

0.33213 

0.33915 

-0.00702 

0.33213 

0.34559 

-0.01346 

165 

0.32447 

0.33460 

-0.01013 

0.32447 

0.36010 

-0.03563 
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Bootstrap 

Sample 


Flyaway 
Std.  Dev. 


Basic 
Airframe 
Std.  Dev. 


Differenced 

Distribution 


Flyaway 
Std.  Dev. 


Other  Air 
Vehicle 
Std.  Dev. 


Differenced 

Distribution 


166 

0.32037 

0.33266 

-0.01229 

0.32037 

0.36659 

-0.04622 

167 

0.31660 

0.33480 

-0.01821 

0.31660 

0.35821 

-0.04162 

168 

0.32574 

0.33931 

-0.01358 

0.32574 

0.35513 

-0.02939 

169 

0.30844 

0.33172 

-0.02328 

0.30844 

0.36539 

-0.05695 

170 

0.31425 

0.33261 

-0.01836 

0.31425 

0.36263 

-0.04837 

171 

0.31516 

0.33920 

-0.02405 

0.31516 

0.36677 

-0.05162 

172 

0.30846 

0.32965 

-0.02120 

0.30846 

0.34936 

-0.04090 

173 

0.31933 

0.34307 

-0.02374 

0.31933 

0.35787 

-0.03854 

174 

0.30425 

0.32601 

-0.02176 

0.30425 

0.36387 

-0.05962 

175 

0.31931 

0.33094 

-0.01163 

0.31931 

0.37014 

-0.05082 

176 

0.31774 

0.34012 

-0.02238 

0.31774 

0.37425 

-0.05652 

177 

0.31132 

0.32747 

-0.01615 

0.31132 

0.36534 

-0.05402 

178 

0.31382 

0.33303 

-0.01921 

0.31382 

0.34550 

-0.03168 

179 

0.31997 

0.34700 

-0.02703 

0.31997 

0.34705 

-0.02707 

180 

0.31540 

0.33203 

-0.01663 

0.31540 

0.36840 

-0.05300 

181 

0.32519 

0.34236 

-0.01717 

0.32519 

0.36081 

-0.03563 

182 

0.31170 

0.33428 

-0.02258 

0.31170 

0.35419 

-0.04249 

183 

0.30592 

0.35539 

-0.04946 

0.30592 

0.35692 

-0.05100 

184 

0.31692 

0.33899 

-0.02207 

0.31692 

0.36264 

-0.04571 

185 

0.31836 

0.33735 

-0.01899 

0.31836 

0.35376 

-0.03540 

186 

0.31830 

0.34194 

-0.02364 

0.31830 

0.35261 

-0.03431 

187 

0.30369 

0.34021 

-0.03652 

0.30369 

0.36737 

-0.06367 

188 

0.31340 

0.33636 

-0.02296 

0.31340 

0.36087 

-0.04748 

189 

0.31423 

0.34639 

-0.03216 

0.31423 

0.37019 

-0.05595 

190 

0.32382 

0.33478 

-0.01096 

0.32382 

0.35952 

-0.03570 

191 

0.31780 

0.34145 

-0.02366 

0.31780 

0.35267 

-0.03488 

192 

0.32166 

0.33170 

-0.01004 

0.32166 

0.35204 

-0.03038 

193 

0.30073 

0.33082 

-0.03009 

0.30073 

0.35909 

-0.05835 

194 

0.32396 

0.33425 

-0.01030 

0.32396 

0.34914 

-0.02518 

195 

0.32501 

0.34170 

-0.01668 

0.32501 

0.36358 

-0.03857 

196 

0.31899 

0.33209 

-0.01310 

0.31899 

0.35922 

-0.04023 

197 

0.32858 

0.33768 

-0.00910 

0.32858 

0.38175 

-0.05317 

198 

0.33390 

0.33117 

0.00273 

0.33390 

0.36473 

-0.03084 

199 

0.32065 

0.32605 

-0.00540 

0.32065 

0.35585 

-0.03520 

200 

0.31518 

0.34611 

-0.03092 

0.31518 

0.36030 

-0.04512 
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Appendix  C:  Bootstrap  Data  for  Inter-theatre  Airlift  Category 


Bootstrap 

Sample 

Flyaway 
Std.  Dev. 

Basic 
Airframe 
Std.  Dev. 

Differenced 

Distribution 

Flyaway 
Std.  Dev. 

Other  Air 
Vehicle 
Std.  Dev. 

Differenced 

Distribution 

1 

0.19824 

0.21628 

-0.01804 

0.1982 

0.2749 

-0.07665 

2 

0.20725 

0.22963 

-0.02238 

0.2072 

0.2796 

-0.07232 

3 

0.19749 

0.22174 

-0.02425 

0.1975 

0.2677 

-0.07024 

4 

0.19079 

0.22375 

-0.03295 

0.1908 

0.2699 

-0.07915 

5 

0.19971 

0.22338 

-0.02366 

0.1997 

0.2736 

-0.07384 

6 

0.20026 

0.22884 

-0.02859 

0.2003 

0.2735 

-0.07322 

7 

0.19705 

0.22876 

-0.03171 

0.1970 

0.2822 

-0.08513 

8 

0.19739 

0.22797 

-0.03058 

0.1974 

0.2634 

-0.06600 

9 

0.20051 

0.22370 

-0.02319 

0.2005 

0.2744 

-0.07390 

10 

0.20073 

0.22658 

-0.02585 

0.2007 

0.2652 

-0.06448 

11 

0.20384 

0.22195 

-0.01811 

0.2038 

0.2719 

-0.06804 

12 

0.19762 

0.22284 

-0.02522 

0.1976 

0.2667 

-0.06911 

13 

0.19859 

0.22985 

-0.03126 

0.1986 

0.2705 

-0.07193 

14 

0.20087 

0.22123 

-0.02036 

0.2009 

0.2854 

-0.08448 

15 

0.20078 

0.22310 

-0.02232 

0.2008 

0.2704 

-0.06960 

16 

0.19987 

0.21869 

-0.01882 

0.1999 

0.2769 

-0.07704 

17 

0.19027 

0.22289 

-0.03262 

0.1903 

0.2861 

-0.09584 

18 

0.20059 

0.23422 

-0.03363 

0.2006 

0.2767 

-0.07615 

19 

0.18703 

0.22152 

-0.03449 

0.1870 

0.2658 

-0.07873 

20 

0.20276 

0.22581 

-0.02305 

0.2028 

0.2766 

-0.07385 

21 

0.18987 

0.22027 

-0.03040 

0.1899 

0.2774 

-0.08751 

22 

0.19153 

0.22689 

-0.03536 

0.1915 

0.2743 

-0.08279 

23 

0.19735 

0.22351 

-0.02616 

0.1973 

0.2658 

-0.06845 

24 

0.20260 

0.22469 

-0.02208 

0.2026 

0.2766 

-0.07395 

25 

0.19680 

0.22039 

-0.02359 

0.1968 

0.2646 

-0.06783 

26 

0.20338 

0.22773 

-0.02435 

0.2034 

0.2687 

-0.06529 

27 

0.19701 

0.22436 

-0.02734 

0.1970 

0.2621 

-0.06507 

28 

0.19565 

0.23187 

-0.03622 

0.1956 

0.2724 

-0.07677 

29 

0.20953 

0.22540 

-0.01588 

0.2095 

0.2845 

-0.07500 

30 

0.19912 

0.22654 

-0.02742 

0.1991 

0.2741 

-0.07494 

31 

0.19539 

0.22434 

-0.02895 

0.1954 

0.2696 

-0.07416 

32 

0.19522 

0.22878 

-0.03356 

0.1952 

0.2732 

-0.07801 

33 

0.19240 

0.22645 

-0.03405 

0.1924 

0.2750 

-0.08260 

34 

0.19978 

0.21820 

-0.01841 

0.1998 

0.2683 

-0.06855 

35 

0.19328 

0.23340 

-0.04013 

0.1933 

0.2776 

-0.08430 

36 

0.19822 

0.21369 

-0.01547 

0.1982 

0.2703 

-0.07211 

37 

0.19301 

0.21891 

-0.02590 

0.1930 

0.2701 

-0.07707 

38 

0.20694 

0.22738 

-0.02043 

0.2069 

0.2718 

-0.06481 

39 

0.19927 

0.21977 

-0.02051 

0.1993 

0.2683 

-0.06899 
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Bootstrap 

Sample 

Flyaway 
Std.  Dev. 

Basic 
Airframe 
Std.  Dev. 

Differenced 

Distribution 

Flyaway 
Std.  Dev. 

Other  Air 
Vehicle 
Std.  Dev. 

Differenced 

Distribution 

40 

0.19194 

0.21821 

-0.02627 

0.1919 

0.2736 

-0.08161 

41 

0.19672 

0.23042 

-0.03370 

0.1967 

0.2699 

-0.07315 

42 

0.20609 

0.22621 

-0.02012 

0.2061 

0.2719 

-0.06579 

43 

0.19733 

0.22483 

-0.02750 

0.1973 

0.2642 

-0.06682 

44 

0.19940 

0.22454 

-0.02514 

0.1994 

0.2650 

-0.06565 

45 

0.19736 

0.22383 

-0.02647 

0.1974 

0.2774 

-0.08004 

46 

0.19586 

0.22854 

-0.03269 

0.1959 

0.2723 

-0.07641 

47 

0.19745 

0.22047 

-0.02303 

0.1974 

0.2711 

-0.07361 

48 

0.19482 

0.22959 

-0.03477 

0.1948 

0.2710 

-0.07618 

49 

0.19726 

0.22582 

-0.02856 

0.1973 

0.2705 

-0.07322 

50 

0.19755 

0.22258 

-0.02503 

0.1976 

0.2717 

-0.07416 

51 

0.20246 

0.22460 

-0.02214 

0.2025 

0.2789 

-0.07647 

52 

0.19520 

0.22424 

-0.02904 

0.1952 

0.2652 

-0.07002 

53 

0.19682 

0.22765 

-0.03082 

0.1968 

0.2679 

-0.07110 

54 

0.20192 

0.23637 

-0.03445 

0.2019 

0.2735 

-0.07161 

55 

0.20005 

0.21405 

-0.01401 

0.2000 

0.2681 

-0.06801 

56 

0.19587 

0.22430 

-0.02843 

0.1959 

0.2684 

-0.07255 

57 

0.20281 

0.22932 

-0.02651 

0.2028 

0.2642 

-0.06138 

58 

0.19308 

0.22682 

-0.03374 

0.1931 

0.2763 

-0.08319 

59 

0.19813 

0.22726 

-0.02913 

0.1981 

0.2695 

-0.07141 

60 

0.19817 

0.22396 

-0.02579 

0.1982 

0.2687 

-0.07055 

61 

0.19664 

0.22935 

-0.03271 

0.1966 

0.2775 

-0.08084 

62 

0.19868 

0.23244 

-0.03376 

0.1987 

0.2684 

-0.06976 

63 

0.20622 

0.22485 

-0.01863 

0.2062 

0.2785 

-0.07225 

64 

0.19932 

0.22864 

-0.02932 

0.1993 

0.2681 

-0.06880 

65 

0.21280 

0.22642 

-0.01362 

0.2128 

0.2692 

-0.05640 

66 

0.20044 

0.22527 

-0.02483 

0.2004 

0.2812 

-0.08081 

67 

0.19872 

0.21992 

-0.02119 

0.1987 

0.2749 

-0.07613 

68 

0.20374 

0.21955 

-0.01581 

0.2037 

0.2693 

-0.06552 

69 

0.19791 

0.23014 

-0.03223 

0.1979 

0.2827 

-0.08477 

70 

0.20065 

0.22665 

-0.02599 

0.2007 

0.2647 

-0.06402 

71 

0.19959 

0.22709 

-0.02750 

0.1996 

0.2725 

-0.07291 

72 

0.18715 

0.23131 

-0.04416 

0.1871 

0.2747 

-0.08752 

73 

0.20935 

0.22598 

-0.01663 

0.2093 

0.2846 

-0.07522 

74 

0.20095 

0.22225 

-0.02129 

0.2010 

0.2739 

-0.07299 

75 

0.19542 

0.23583 

-0.04041 

0.1954 

0.2713 

-0.07592 

76 

0.20148 

0.21982 

-0.01834 

0.2015 

0.2782 

-0.07671 

77 

0.20537 

0.22645 

-0.02107 

0.2054 

0.2701 

-0.06476 

78 

0.19700 

0.22722 

-0.03022 

0.1970 

0.2732 

-0.07625 

79 

0.20252 

0.22943 

-0.02691 

0.2025 

0.2700 

-0.06743 

80 

0.20403 

0.22123 

-0.01719 

0.2040 

0.2745 

-0.07051 

81 

0.19423 

0.23113 

-0.03690 

0.1942 

0.2737 

-0.07946 

Ill 


Bootstrap 

Sample 

Flyaway 
Std.  Dev. 

Basic 
Airframe 
Std.  Dev. 

Differenced 

Distribution 

Flyaway 
Std.  Dev. 

Other  Air 
Vehicle 
Std.  Dev. 

Differenced 

Distribution 

82 

0.19659 

0.23081 

-0.03422 

0.1966 

0.2710 

-0.07445 

83 

0.20502 

0.22191 

-0.01689 

0.2050 

0.2718 

-0.06678 

84 

0.19576 

0.23380 

-0.03804 

0.1958 

0.2617 

-0.06590 

85 

0.19289 

0.22265 

-0.02976 

0.1929 

0.2731 

-0.08016 

86 

0.19386 

0.22276 

-0.02891 

0.1939 

0.2691 

-0.07529 

87 

0.19391 

0.22309 

-0.02918 

0.1939 

0.2740 

-0.08012 

88 

0.19479 

0.21983 

-0.02504 

0.1948 

0.2669 

-0.07216 

89 

0.19014 

0.21794 

-0.02780 

0.1901 

0.2745 

-0.08432 

90 

0.19391 

0.22055 

-0.02665 

0.1939 

0.2739 

-0.07996 

91 

0.18817 

0.23294 

-0.04478 

0.1882 

0.2828 

-0.09459 

92 

0.19742 

0.22451 

-0.02708 

0.1974 

0.2774 

-0.08000 

93 

0.19978 

0.23027 

-0.03049 

0.1998 

0.2590 

-0.05919 

94 

0.19566 

0.22635 

-0.03070 

0.1957 

0.2736 

-0.07792 

95 

0.20610 

0.21894 

-0.01284 

0.2061 

0.2811 

-0.07501 

96 

0.19566 

0.21674 

-0.02108 

0.1957 

0.2750 

-0.07937 

97 

0.19864 

0.23823 

-0.03959 

0.1986 

0.2648 

-0.06613 

98 

0.20037 

0.22802 

-0.02765 

0.2004 

0.2761 

-0.07576 

99 

0.19914 

0.22711 

-0.02797 

0.1991 

0.2729 

-0.07381 

100 

0.19520 

0.22499 

-0.02979 

0.1952 

0.2728 

-0.07755 

101 

0.20498 

0.22824 

-0.02326 

0.2050 

0.2774 

-0.07246 

102 

0.20554 

0.22518 

-0.01965 

0.2055 

0.2762 

-0.07062 

103 

0.19135 

0.22527 

-0.03392 

0.1914 

0.2720 

-0.08061 

104 

0.20336 

0.21894 

-0.01558 

0.2034 

0.2666 

-0.06323 

105 

0.20152 

0.22103 

-0.01951 

0.2015 

0.2702 

-0.06865 

106 

0.19369 

0.22877 

-0.03508 

0.1937 

0.2730 

-0.07932 

107 

0.19391 

0.22109 

-0.02718 

0.1939 

0.2763 

-0.08234 

108 

0.20069 

0.22654 

-0.02585 

0.2007 

0.2769 

-0.07624 

109 

0.20342 

0.21376 

-0.01033 

0.2034 

0.2769 

-0.07344 

110 

0.20196 

0.23004 

-0.02808 

0.2020 

0.2729 

-0.07090 

111 

0.19953 

0.22721 

-0.02768 

0.1995 

0.2676 

-0.06802 

112 

0.20047 

0.23483 

-0.03436 

0.2005 

0.2756 

-0.07509 

113 

0.19237 

0.22638 

-0.03401 

0.1924 

0.2759 

-0.08350 

114 

0.19706 

0.22717 

-0.03012 

0.1971 

0.2653 

-0.06827 

115 

0.20541 

0.23094 

-0.02554 

0.2054 

0.2637 

-0.05830 

116 

0.19985 

0.22499 

-0.02514 

0.1998 

0.2731 

-0.07323 

117 

0.20292 

0.21754 

-0.01463 

0.2029 

0.2777 

-0.07474 

118 

0.19264 

0.21900 

-0.02636 

0.1926 

0.2709 

-0.07827 

119 

0.19676 

0.23047 

-0.03371 

0.1968 

0.2737 

-0.07694 

120 

0.19994 

0.22120 

-0.02126 

0.1999 

0.2574 

-0.05741 

121 

0.20153 

0.23212 

-0.03059 

0.2015 

0.2689 

-0.06735 

122 

0.19803 

0.23073 

-0.03270 

0.1980 

0.2748 

-0.07675 

123 

0.20125 

0.22431 

-0.02307 

0.2012 

0.2725 

-0.07130 
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Bootstrap 

Sample 

Flyaway 
Std.  Dev. 

Basic 
Airframe 
Std.  Dev. 

Differenced 

Distribution 

Flyaway 
Std.  Dev. 

Other  Air 
Vehicle 
Std.  Dev. 

Differenced 

Distribution 

124 

0.19261 

0.22417 

-0.03156 

0.1926 

0.2701 

-0.07748 

125 

0.20059 

0.22962 

-0.02903 

0.2006 

0.2762 

-0.07563 

126 

0.20549 

0.22265 

-0.01716 

0.2055 

0.2622 

-0.05671 

127 

0.19634 

0.22674 

-0.03040 

0.1963 

0.2695 

-0.07313 

128 

0.20526 

0.22800 

-0.02274 

0.2053 

0.2728 

-0.06758 

129 

0.19984 

0.23076 

-0.03092 

0.1998 

0.2627 

-0.06285 

130 

0.19589 

0.23255 

-0.03665 

0.1959 

0.2784 

-0.08254 

131 

0.19600 

0.23915 

-0.04315 

0.1960 

0.2608 

-0.06482 

132 

0.20537 

0.22213 

-0.01676 

0.2054 

0.2653 

-0.05990 

133 

0.20182 

0.22819 

-0.02637 

0.2018 

0.2791 

-0.07724 

134 

0.19028 

0.22661 

-0.03633 

0.1903 

0.2839 

-0.09366 

135 

0.19889 

0.22610 

-0.02721 

0.1989 

0.2679 

-0.06906 

136 

0.19448 

0.22569 

-0.03121 

0.1945 

0.2778 

-0.08329 

137 

0.19672 

0.23372 

-0.03700 

0.1967 

0.2807 

-0.08400 

138 

0.20493 

0.23046 

-0.02553 

0.2049 

0.2648 

-0.05985 

139 

0.19968 

0.22663 

-0.02695 

0.1997 

0.2711 

-0.07145 

140 

0.19552 

0.21780 

-0.02228 

0.1955 

0.2723 

-0.07679 

141 

0.20087 

0.22340 

-0.02253 

0.2009 

0.2590 

-0.05811 

142 

0.19628 

0.23067 

-0.03439 

0.1963 

0.2631 

-0.06682 

143 

0.19930 

0.22899 

-0.02970 

0.1993 

0.2763 

-0.07704 

144 

0.20486 

0.22231 

-0.01746 

0.2049 

0.2687 

-0.06389 

145 

0.20278 

0.22265 

-0.01987 

0.2028 

0.2685 

-0.06569 

146 

0.19554 

0.23131 

-0.03577 

0.1955 

0.2799 

-0.08438 

147 

0.19849 

0.22825 

-0.02976 

0.1985 

0.2745 

-0.07603 

148 

0.19836 

0.22998 

-0.03162 

0.1984 

0.2781 

-0.07969 

149 

0.20577 

0.22370 

-0.01794 

0.2058 

0.2759 

-0.07015 

150 

0.19736 

0.21974 

-0.02238 

0.1974 

0.2693 

-0.07194 

151 

0.20085 

0.21859 

-0.01775 

0.2008 

0.2734 

-0.07259 

152 

0.18903 

0.22506 

-0.03603 

0.1890 

0.2616 

-0.07255 

153 

0.20074 

0.21915 

-0.01841 

0.2007 

0.2668 

-0.06611 

154 

0.19484 

0.21927 

-0.02444 

0.1948 

0.2680 

-0.07319 

155 

0.20279 

0.22058 

-0.01779 

0.2028 

0.2758 

-0.07301 

156 

0.20475 

0.22761 

-0.02285 

0.2048 

0.2584 

-0.05369 

157 

0.21509 

0.22215 

-0.00706 

0.2151 

0.2736 

-0.05848 

158 

0.19603 

0.21558 

-0.01955 

0.1960 

0.2738 

-0.07776 

159 

0.19843 

0.22439 

-0.02596 

0.1984 

0.2716 

-0.07319 

160 

0.19661 

0.23493 

-0.03832 

0.1966 

0.2705 

-0.07386 

161 

0.20237 

0.23627 

-0.03390 

0.2024 

0.2772 

-0.07479 

162 

0.19984 

0.22214 

-0.02230 

0.1998 

0.2696 

-0.06973 

163 

0.19692 

0.22551 

-0.02858 

0.1969 

0.2610 

-0.06410 

164 

0.20170 

0.22325 

-0.02155 

0.2017 

0.2766 

-0.07486 

165 

0.20187 

0.23018 

-0.02830 

0.2019 

0.2657 

-0.06381 
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